terms and definitions used by UK organisations and selected ...

Systematic and other reviews: terms and definitions used by UK

organisations and selected databases

Systematic review and Delphi survey

Lesley Sander, Hilary Kitcher

Support Unit for Research EvidenceWales College of Medicine, Biology, Life and Health Sciences

Cardiff University

This review was undertaken for the Health Development Agency (HDA) but published after the functions of the

HDA were transferred to NICE on 1 April 2005. This document does not represent NICE guidance.

This document is also published on the NICE website at:

www.publichealth.nice.org.uk

ISBN: 1-84629-149-6

© National Institute for Health and Clinical Excellence February 2006

Mayumi Ashizawa, EPPI-CentreDouglas Badenoch, MinervationNick Black, London School of Hygiene and Tropical

MedicineBengt Brorsson, Swedish Council on Technology

Assessment in Health CarePhil Callan, National Health and Medical Research

Council, Australian Government Iain Chalmers, co-founder, Cochrane ColloborationMelanie Corris, National Coordinating Centre for Health

Technology AssessmentMary Dixon-Wood, Cochrane Qualitative Methods

NetworkBenjamin Djulbegovic, H Lee Moffit Centre and Research

Institute, University of FloridaJulie Glanville, Centre for Reviews and DisseminationIan Graham, School of Nursing, University of OttawaTrish Greenhalgh, Unit for Evidence-Based Practice and

Policy, University College LondonAlex Haig, NHS Education, ScotlandRobin Harbour, SIGNJini Hetherington, Cochrane Collaboration SecretariatDoug Jameson, Centre for Health Services and Policy

Research, University of British ColumbiaKhalid Khan, Education Resource Centre,

Birmingham’s Women HospitalKate Light, Centre for Reviews and DisseminationAllyson Lipp, School of Care Sciences, University of

GlamorganKathleen Lohr, RM International, University of North

Carolina Evidence-based Practice CenterAntti Malmivaara, Finnish Office for Health Technology

AssessmentAlain Mayhew, Cochrane Effective Practice and

Organisation of Care GroupSandra Micucci, The Effective Public Health Practice

ProjectDavid Moher, Chalmers Research Group, Children’s

Hospital of Eastern Ontario Research InstituteSarah Moore, The UK Cochrane CentreLisa Morton, Centre for Outcomes Research and

EffectivenessIan Needleman, International Centre for Evidence-

based Periodontal HealthAnne Oakley, EPPI-Centre

Madhukar Pai, Cochrane Collaborative Review Group on HIV Infection and AIDS

Elizabeth Paulsen, Cochrane Methodology Review Group

Jennie Popay, Cochrane Qualitative Methods GroupMarina Rappoport, National Library of Medicine, National

Institutes of HealthRebecca Rees, EPPI-CentreHannah Salvidge, National Institute for Health and

Clinical ExcellencePhilip Satherley, Nursing, Health and Social Care Research

Centre, Wales College of Medicine, Biology, Life and Health Sciences

Rachel Shaw, Department of Health Sciences, University of Leicester

Bev Shea, CIET InternationalJonathan Shepherd, Wessex Institute for Health Research

and Development, University of SouthamptonPhillip Simons, National Coordinating Centre for Health

Technology AssessmentFay Sullivan, ESRC UK Centre for Evidence-based Policy

and Practice, King’s College LondonSupport Unit for Research Evidence, Wales College of

Medicine, Biology, Life and Health SciencesBrian Taylor, Department of Social Work, University

of UlsterHilary Thompson, MRC Social and Public Health

Sciences UnitDavid Tranfield, Cranfield School of ManagementJeffrey Valentine, Department of Psychology: Social and

Health Sciences and Program in Education, Duke University

Arianne Verhagen, Erasmus Medical Centre, University of Rotterdam

Sam Vincent, BazianKaren Visser, Health Care Efficiency Research, ZonMw

(Netherlands Organisation for Health Research and Development)

Alison Weightman, Cardiff UniversityRick Wiechula, Joanna Briggs InstituteSE Wilcock, Joanna Briggs Collaborating Centre for

Evidence-based Multi-professional PracticeKath Wright, Centre for Reviews and Dissemination

ii Systematic and other reviews: terms and definitions used by UK organisations and selected databases Systematic review and Delphi survey

We would like to thank the following people for their assistance and generous contribution throughout the review process.

Acknowledgements

This document is one in a series of occasional papers thatwere originally commissioned and published by the HealthDevelopment Agency (HDA) and were intended to be ofscientific interest to academics and the wider public healthcommunity. In April 2005 the HDA’s functions were takenon by the National Institute for Health and ClinicalExcellence (NICE). This document was planned andcommissioned by the HDA but published after itsfunctions were transferred to NICE; as such, thisdocument does not represent NICE guidance.

The HDA was established in 2000. It was engaged, amongother things, in work intended to explore and developmethodologies to develop the evidence base for publichealth. Within the stream of methodological developmentwork, the HDA commissioned several occasional papers(of which this is one) to review and expand methodologiesrelevant to public health, and to present them in waysthat would be meaningful and useful to those working inresearch – not only academics, but also commissioners,managers and researchers. NICE would be pleased to hearfrom readers who would like to comment on mattersrelating to the content of this paper. Comments can bemade on the NICE website: www.publichealth.nice.org.uk

Professor Michael P. Kelly Director of the Centre for Public Health ExcellenceNational Institute for Health and Clinical Excellence

iiiSystematic and other reviews: terms and definitions used by UK organisations and selected databases Systematic review and Delphi survey

Foreword

iv Systematic and other reviews: terms and definitions used by UK organisations and selected databases Systematic review and Delphi survey

Contents

Acknowledgements ii

Foreword iii

Summary 1

Introduction 4

Methods 6Literature search 6Result of literature selection process 8Delphi survey methods 9

Results – literature review 10Characteristics of included publications 10Data analysis 10

Delphi survey results 14

Discussion 16Limitations of this review 18Conclusions 18

References 19

Appendix 1: Survey of review organisations and database providers 30

Appendix 2: Search terms 33

Appendix 3: Some UK and international organisations 34undertaking systematic reviews

Appendix 4: Data extraction form 35

Appendix 5: Survey of review organisations and database providers 39– update September 2004

Appendix 6: Excluded studies 42

Appendix 7a: Systematic review terms and definitions 46– Delphi survey questionnaire 1

Appendix 7b: Systematic review terms and definitions 47– summary of first Delphi round

Appendix 8a: Systematic review terms and definitions 53– Delphi survey questionnaire 2

Appendix 8b: Systematic review terms and definitions 55– summary of second Delphi round

1Systematic and other reviews: terms and definitions used by UK organisations and selected databases Systematic review and Delphi survey

Summary

Background

The proliferation of systematic literature reviews duringthe past decade has prompted the need to examine thescientific quality of the methodological processes used toundertake and report these reviews. Although guidancefor the undertaking of systematic reviews is currentlyavailable (Centre for Reviews and Dissemination 2001;Green and Higgins 2004), and there is some agreementabout what quality components define a systematicreview (Moher et al. 1999; Oxman and Guyatt 1991;Oxman et al. 1991), there remains a lack of consensus onhow these quality components are then implemented inpractice.

In 2003 a survey of the terminology and definitions of thedifferent types of review article used by five UKorganisations conducting systematic reviews, and somemedical and non-medical database providers of relevanceto public health, was conducted by the Health EvidenceBulletins Wales project team, in collaboration with theHealth Development Agency (HDA – now the NationalInstitute for Health and Clinical Excellence, NICE). Theresults showed that there was little consensus amongdatabase providers on the terms and definitions used todescribe different types of review article. There was aconsensus among the surveyed UK organisations thatthere are three main types of review: systematic review,literature review and consensus group statement.

Based on these findings it was agreed, and endorsed bysurveyed organisations, that it would be helpful to seekthe views of a larger number of UK organisations on aconsensus agreement of review types. The HDAcommissioned the Health Evidence Bulletins Wales projectteam to undertake a systematic literature review toidentify and summarise the principal quality componentsof a systematic review. The findings of the literature

review were then used to design a series of structuredquestions as part of a Delphi consensus survey circulatedto UK organisations currently involved in the undertakingof systematic reviews.

Objective

The objective of this review is to identify – from existingsystematic review guidelines and checklists for evaluatingsystematic reviews – the key components that comprise amethodologically sound systematic review.

Methods

Data sourcesElectronic database searches formed the basis of theliterature search and the following databases weresearched between 1995 and June 2004 for all languages:

• ASSIA (Applied Social Sciences Index and Abstracts) • CareData • CINAHL (Cumulative Index of Nursing and Allied Health

Literature) • Cochrane Library • EMBASE • ERIC (Educational Resources Information Center) • HDA Evidence Base • HDA HealthPromis • HMIC (Health Management Information Consortium) • Internet • ISI Social Science Citation Index • ISI Science Citation Index • MEDLINE • MEDLINE In-Process & Other Non-Indexed Citations • NRR (National Research Register) • PsycInfo

• SIGLE (System for Information on Grey Literature) • Sociological Abstracts• ZETOC (British Library Electronic Table of Contents).

Additional searches included contact with experts,scanning reference lists of included publications, and afurther electronic database search of the most frequentlycited authors. Handsearching of relevant journals was notconducted.

Inclusion criteriaThis review considered all publications that:

• concerned the methods or provided guidance forundertaking a systematic review

• evaluated the quality of a systematic review • reviewed systematic reviews• concerned the indexing or keywording of a systematic

review in an electronic database.

These criteria were applied to all publications thatdiscussed a systematic review, meta-analysis, overview,research synthesis, health technology assessment orhealth impact assessment.

Selection of publicationsCitations identified by the electronic literature searchwere imported into reference management software andscreened by one reviewer for relevance, based on theinclusion criteria. The titles and abstracts of citations wereexamined for relevance and where a decision could notbe made on the basis of title and/or abstract thepublication was retrieved in full.

Data abstractionA data extraction form was developed based on all of thepotential quality criteria that may define a systematicreview; potential criteria were pooled from existingcriteria listings and established guidelines. The formcomprises ten primary criteria questions with a total of 72 constituent components.

Quality assessmentAs this review was an assessment of authors’ views andopinions, it was not considered appropriate to assess thequality of the included publications.

Data synthesisData were abstracted and results were compiled byfrequency counting, in an attempt to quantify current

guidance. Percentage ratings were calculated based onthe total frequency score for each quality criterion and itscomponents.

Delphi surveyThose quality criteria that were most frequently cited bythe included methodological guidance were then used toform the basis of a series of structured questions as partof a Delphi consensus survey. Survey questionnaires werecirculated to UK organisations undertaking systematicreviews to elicit their views and assessments, and finallyto arrive at a consensus.

Survey updateThe original survey of some UK organisations and(international) database providers, conducted in mid-2003, was updated in September 2004 to confirm anychanges in the ‘review’ terms identified during the pilotsurvey. The same methods were used in both surveys, buttwo additional databases (CINAHL and HMIC) wereadded, as systematic review terms had been identified asindexed subject headings for these databases during theperiod between surveys.

Ad hoc database searchDuring the writing of the ‘Discussion’ section an ad hocsearch was conducted on PubMed/BioMed Central andMEDLINE. It had been assumed that PubMed and MEDLINEindexed identical content, but these support searches,seeking evidence of ‘protocol’ or ‘years searched’ ascriteria, identified a further seven relevant papers onPubMed that had not been retrieved on MEDLINE. As aresult the authors of this review, when undertaking asystematic review in future, will search PubMed in additionto MEDLINE. The seven additional papers were included inthe review, despite their late identification.

Results

Literature reviewA total of 191 publications, published between 1995 and2004, satisfied the inclusion criteria.

• The reporting of a literature search strategy foridentifying potentially relevant literature wasconsidered to be the most critical element in thesystematic review process – 97% of publications statedthat reviewers should verify that they had conducted asystematic literature search.

2 Systematic and other reviews: terms and definitions used by UK organisations and selected databases Systematic review and Delphi survey

• Ninety-one per cent stated that the synthesis of data tointegrate the results of all included studies was thenext most important quality criteria. This was nearlymatched by the need for data to be synthesised (90%),the assessment of study quality (87%) and theinclusion of predetermined written criteria for theselection of studies (87%).

• The methods for extracting data (74%) and selectingstudies for inclusion (71%) were viewed to be the nextmost significant criteria when conducting a systematicreview.

• Forty-nine per cent of publications reported the needfor a pre-defined study protocol.

• In terms of good reporting practice, less than half(33%) described the structure and/or components ofthe final report and 23 papers (12%) discussed therationale for undertaking a systematic review.

Delphi surveyThe Delphi survey was sent out by email to 27organisations and received 20 replies with 16 fullycompleted questionnaires.

• Thirteen out of 16 agreed that there should be a UK-wide consensus for the definition of a systematicreview, although three were strongly against this andreported the reasons for their objection.

• Nine quality components were either approvedunanimously (four criteria) or by a great majority (14 or15 out of 16). The lowest level of agreement was 13out of 16 for the inclusion of a protocol.

Based on these findings, the second Delphi survey wassent out to the 16 who had replied to stage 1. This had11 replies with 10 completed questionnaires.

• The majority (9 out of 10) supported the view that adefinition of ‘a systematic review’ should be agreed foruse in the UK, but should also meet the definitionsused worldwide by the Cochrane and Campbellcollaborations. Only one preferred a definition to beagreed for use within the UK only.

• The list of nine quality criteria was extended by afurther five, suggested by the first stage replies, and areworded version was offered for all criteria that hadnot been unanimously approved during the first round.Preferences were evenly matched between those whopreferred the original wording and those who favouredthe new version. The majority approved of theadditional items (7 or 8 out of 10) with two being

against the idea of a standardised list of potentialbiases.

Limitations

The literature review part of this study was conducted bya single reviewer and key data may have been missed ormisinterpreted during data extraction. Includedpublications were not always explicit in their guidanceand steps in the review process were reported withvarying strength and emphasis. This may have resulted inan item receiving a score by one reviewer that may havenot been selected by another. As there was no secondreviewer this could not be qualified by inter-rateragreement. The concept of quality was not examined inthis review but was interpreted to signify a definingmeasure or standard that was repeatedly cited in theincluded papers.

Conclusions

The results of this literature review show that there issome agreement regarding the ‘primary’ quality criteriafor a systematic review and that there is an emerginghierarchical structure for agreement about the principalelements that define a systematic review. The results ofthe Delphi survey further confirm agreement of thehighlighted criteria components. It is this agreement orconsensus that may ultimately determine an establishedand approved definition of a systematic review forapplication in the UK.


Systematic literature reviews and meta-analyses haveassumed an increasingly central role in healthcareliterature during the past decade (Ernst and Pittler 2001;Grootens et al. 2003). Systematic reviews of all relevantresearch about the efficacy of a treatment areincreasingly seen by clinicians, healthcare providers andfunding agencies as providing the most reliable basis forconclusions about treatment effects (Cook et al. 1992;Forbes 2003; Gray 1997). The value of systematic reviewsis also recognised by practitioners and academics outsidehealthcare, with review articles published in the fields ofeducation, criminology, management knowledge and thesocial and political sciences (Macdonald 2003; Oakley2002; Petrosino et al. 2003; Petticrew 2005).

This proliferation of review articles has prompted theneed to examine the scientific quality of themethodological processes used to undertake and reportthem. Comparative critical evaluation studies (Jadad et al.2000; Shea et al. 2002) have assessed the quality ofpublished reviews and validated checklists have beendeveloped to assess their quality (Oxman and Guyatt1991; Shea et al. 2001; West et al. 2002). Severalreviewers have provided guidelines for reporting meta-analyses (Cook et al. 1995; Stroup et al. 2000). One ofthe most recent developments has been the Quality ofReporting of Meta-analyses (QUOROM) conference wherean evidence-based criteria checklist for reportingstandards was developed (Moher et al. 1999). This iscurrently being used and evaluated (Hemels et al. 2004)as a reporting standard. The British Medical Journal nowstipulates use of the QUOROM checklist and flowchart byauthors reporting systematic reviews of randomisedcontrolled trials and provides these frameworks on itswebsite. Other journals are following this trend.

As the number of review articles has increased, so has thenumber of guidelines on how to undertake a systematic

literature review. Guidance, however, has been diverseand although there is some agreement about whatquality components define a systematic review, there is alack of consensus on how these quality components areimplemented in practice. For this reason publishedguidance has varied in emphasis on what critical featuresdistinguish systematic reviews from the non-systematicliterature. Guideline and review authors themselves oftenseem uncertain about the terminology and processes thatdefine different review types. Review articles that claim tobe systematic vary in quality and may appear to havebeen conducted with less rigour than some reviews thatdo not claim to be systematic.

In 2003 the Health Evidence Bulletins Wales project team,in collaboration with the Health Development Agency(HDA – now the National Institute for Health and ClinicalExcellence, NICE), completed a survey of the terminologyand definitions of the different types of review articleused by five UK organisations conducting systematicreviews and also by the providers of some medical andnon-medical databases of relevance to public health.Survey findings showed that there was little consensusamong database providers on the terms and definitionsused to describe different types of review article. Incontrast, a consensus emerged within UK organisationsthat there were three main types of review: systematicreview, literature review and consensus group statement(Appendix 1). Based on these findings it was agreed, andendorsed by surveyed organisations, that it would behelpful to seek the views of a larger number of UKorganisations to establish a consensus agreement ofreview types.

The HDA commissioned the Health Evidence BulletinWales project team to undertake this systematic literaturereview of views on the quality criteria for systematicreviews, including research on the quality of Cochrane


Introduction

and other systematic reviews and process guidelines. Theobjective of this review is, first, to identify from existingsystematic review guidelines and checklists for evaluatingsystematic reviews the key components that comprise amethodologically sound systematic review. The identifiedcriteria components were then used to design a series ofstructured questions as part of a Delphi consensus surveythat was circulated to UK organisations currently involvedin the undertaking of systematic reviews.


Literature search

Search strategyAn electronic scoping search was undertaken onMEDLINE and EMBASE to assess the scale and range ofpotentially relevant papers, to identify lead authors andkey papers, and to analyse indexed search terms andsubject headings used in key papers.

Following the scoping search, further trial searches wereundertaken for the purpose of developing the searchstrategy. Indexed search terms and subject headings fromkey papers were further tested to identify additionalsubject headings. The review topic was broken down intomultiple terms and variations of these terms wereexamined to test the ability of the terms to retrieve themost relevant papers. Based on the outcome of these trialsearches a broad search strategy was developed for ahigh sensitivity recall (Appendix 2).

Databases The following electronic databases, identified as relevantto the review topic, were systematically searchedbetween 1995 and June 2004 for all languages:

• ASSIA (Applied Social Sciences Index and Abstracts) • CareData • CINAHL (Cumulative Index of Nursing and Allied Health

Literature) • Cochrane Library • EMBASE • ERIC (Educational Resources Information Center) • HDA Evidence Base • HDA HealthPromis • HMIC (Health Management Information Consortium) • Internet • ISI Social Science Citation Index • ISI Science Citation Index

• MEDLINE • MEDLINE In-Process & Other Non-Indexed Citations • NRR (National Research Register) • PsycInfo • SIGLE (System for Information on Grey Literature) • Sociological Abstracts• ZETOC (British Library Electronic Table of Contents).

A start date of 1995 was selected for searching, asguidance for the undertaking of systematic reviews didnot appear in the published literature much before then.

Contact with experts Organisations involved in conducting systematic reviewswere identified through discussion with colleagues and asearch of the Internet using a general search engine.Hyperlinks on the home pages of systematic revieworganisations were investigated to identify additionalorganisations. Key authors identified during the scopingexercise were contacted directly and an informationrequest was posted on the following JISCmail discussionlists: EVIDENCE-BASED HEALTH, SYS-REVIEW, PUBLIC-HEALTH and LIS-MEDICAL.

A letter describing the review topic was sent to allorganisations identified asking them to suggest anyresearch or published literature looking at theterminology, definitions and/or quality criteriacomponents used for review literature (Appendix 3). All replies were methodically followed up.

Additional searchesResults of the electronic literature searches were importedinto the bibliographical management software, ReferenceManager, and duplicates were deleted. After referenceswere selected on the basis of title and abstract forpotential inclusion, a search was conducted to identify allauthors who were cited three or more times from the


Methods

retained studies. An electronic author search was then runon the MEDLINE and EMBASE databases to identify anyfurther relevant studies by these authors. The referencelists of all included publications were scanned during dataabstraction for any additional papers meeting the inclusioncriteria. Handsearching of relevant journals was notconducted due to the restricted timescale of the review.Similarly, an update electronic search was not conducted.

Inclusion/exclusion criteriaDevelopment of the inclusion and exclusion criteria wasinformed by the results of the initial scoping search anddiscussion between the HDA and the reviewer (LesleySander). See Table 1 for full inclusion/exclusion criteria.

Terms and definitionsThe broader definition of the term ‘published’, meaning‘information in the public arena’, was used in the context

of this review. Web-based, electronic publications andpublications currently in press were included. It ispossible, therefore, that some publications were not peer-reviewed and may have been accessed from websites thathave not been evaluated. The term ‘agreement’ usedthroughout this review has no numerical interpretationbut was used to indicate compatibility of observations.

Selection of publicationsCitations identified by the electronic literature searchwere imported into Reference Manager and screened bythe reviewer in a stepped process based on the pre-defined inclusion and exclusion criteria. The titles andabstracts of citations were screened for relevance andwhere a decision could not be made on the basis of titleand abstract the publication was retrieved in full.


Inclusion criteria

Paper to be included if it meets the following criteria:

1 Concerns the methods or provides a guide for undertaking a systematic review*

2 Evaluates or monitors the quality of a systematic review3 Evaluates the quality of reporting standards of a systematic

review4 Concerns the methods or provides a guide for undertaking a

meta-analysis, quantitative review or economic evaluation supported by a systematic review

5 Reviews systematic reviews (eg an evidence briefing paper conducted by the Health Development Agency)

6 Identifies and/or evaluates a quality or critical appraisal checklist to appraise systematic reviews

7 Concerns the indexing or keywording of a systematic review or meta-analysis in an electronic database

8 Critiques a Cochrane Collaboration or other systematic review9 Compares a Cochrane Collaboration review with any other

published systematic review

Exclusion criteria

Paper to be excluded if it meets the following criteria:

1 Concerns a clinical guideline (although these may be based on a systematic review methodology they have been excludeddue to the large number of guidelines currently available)

2 Concerns a process guideline for a clinical guideline (unless the abstract clearly indicates that the guideline included a methodology for the systematic review to support the guideline development)

3 Solely concerns any of the individual components of a systematic review; eg the inclusion of trials published in languages other than English or the inclusion of grey literature; literature search methods; variation in reporting of summary measures; or analyses used to pool statistical data

4 Solely concerns any of the individual components of meta-analyses or quantitative analyses; eg a comparison or evaluation of statistical analyses

5 Evaluates or monitors the quality of a randomised controlled trial or any other primary research design; eg papers that discuss CONSORT or CONSORT guidelines

6 Concerns the quality of reporting of a randomised controlled trial

7 Concerns the quality of or quality of reporting of an abstract of a systematic review

8 Concerns the application of systematic review-level evidence in clinical practice or a clinical setting

Table 1: Inclusion/exclusion criteria

Limits: Human, all languages, all countries, 1995–2004, all publication types

*For systematic review, read review or overview or research synthesis or health technology assessment or health impact assessment.

Data extraction A data extraction form was developed based on all of thepotential quality criteria that may define a systematicreview. Potential criteria were gathered from existingcriteria listings and established guidelines (Centre forReviews and Dissemination 2001; Green and Higgins2004; Oxman et al. 1991). The form comprises tenprimary criteria questions with a total of 72 constituentcomponents (Appendix 4). A draft data extraction formwas piloted on five papers. As no changes to the formresulted from the pilot it was accepted and usedthroughout the data extraction.

Quality assessmentAs this review was an assessment and summary ofauthors’ views and opinions, it was not consideredappropriate to assess the quality of the includedpublications.

Survey updateThe original survey (Appendix 1) of some UKorganisations carrying out systematic reviews and(international) database providers, conducted in mid-2003, was updated in September 2004 to confirm anychanges in the ‘review’ terms identified during the pilotsurvey (Appendix 5). The same methods were usedduring the update survey as those used during theoriginal survey, with the inclusion of two additionaldatabases (CINAHL and HMIC), as systematic reviewterms had been identified as indexed subject headings forthese databases during the period between surveys.

Ad hoc database searchDuring the writing of the ‘Discussion’ section an ad hocsearch was conducted on PubMed/BioMed Central andMEDLINE. It had been assumed that PubMed andMEDLINE indexed identical content, but these supportsearches, seeking evidence of ‘protocol’ or ‘yearssearched’ as criteria, identified a further seven relevantarticles on PubMed which had not been retrieved onMEDLINE. As a result the authors of this review willsearch PubMed/BioMed Central in addition to MEDLINE.when carrying out future systematic reviews. The sevenadditional papers were included in the review, despitetheir late identification.

Result of literature selection process

A total of 2993 citations was identified by the electronicdatabase searches. These were imported into ReferenceManager software and duplicates were eliminated. Thecitations were then screened by one reviewer, applyinginclusion criteria for eligibility based on title and abstract.When both duplicates and ineligible citations wereexcluded, 350 citations remained. After the full text wasretrieved for all remaining citations, a further 138publications were excluded on the basis of the inclusionand exclusion criteria. A ‘top ten’ author search wasconducted on the remaining 212 eligible publications andnine additional papers were identified – so 221 citationswere retained for data abstraction.

During data abstraction a further seven citations wereidentified from reference lists, and two publications thatwere in progress were made available by the authors.Overall, 42 papers were excluded during data abstractionas they did not satisfy the inclusion criteria or systematicreview process guidance was not reported. (See Appendix6 for reasons for exclusions.)

A total of 184 publications published between 1995 and2004 were included for data analysis (four citations couldnot be obtained). Although the abstract for onepublication was available, its table providing thefrequency of key quality features reported in the includedstudies was not. Attempts to obtain publications weremade until data abstraction was completed. Onepublication was provided by a colleague after datacollation had been completed (Pawson et al. in press).As this publication was not publicly available, butprovided useful and usable data, it was included and thedata were integrated into the final results. One relevantpublication was not included as it is currently in draftformat only (Jackson and Waters). Seven further paperswere added as a result of the additional ad hoc PubMedsearch.

The final selection comprised 191 publications – see‘Included publications’ in the ‘References’ section, p21. Of these, second papers were published for fourpublications and six papers were part of two separateseries. One data abstraction sheet was completed foreach of the combined publications. When an updatedpublication was available, only the updated version wasincluded.


Delphi survey methods

The Delphi survey method was chosen to enable a widegroup of interested organisations and parties tocontribute to this study (McBride et al. 2003). The Delphiconsensus technique uses a series of questionnaires thatare completed and returned anonymously. Responses tothe first questionnaire are analysed and collated; a secondquestionnaire is then sent out including full feedback onthe stage 1 responses and giving a smaller set ofquestions that invite participants to comment on thefindings so far. It is possible to continue this exercise forseveral stages, but in practice the process is considered tobe complete once a consensus has been reached or whensufficient information exchange has been obtained(Delbecq et al. 1975; McBride et al. 2003). Consensus hasbeen defined as ‘general agreement of a substantialmajority’ (Verhagen et al. 1998). The key features of theDelphi technique are anonymity, iteration, controlledfeedback and statistical aggregation of group response(Rowe and Wright 1999).

For this survey there were two Delphi rounds.Questionnaires were emailed to 27 organisations andnamed parties who had been involved in the earlierstages of this investigation. The first questionnaire(Appendix 7a) invited opinions on the need for a UK-wideconsensus on definitions for ‘a systematic review’ andvarious related terms. It also asked respondents forratings of the nine most frequently cited quality criteriacomponents for reporting a systematic review. Criteriacomponents were based on those identified during theanalysis of the results of the literature review (see Table 3,p11, for the list of cited quality criteria).

The detailed feedback from this provided someinteresting suggestions and insights (Appendix 7b). An analysis of the feedback, including all responses, wasthen emailed back to the 16 who responded, togetherwith a stage 2 questionnaire based on their replies.Opinions were sought on the need for a ‘systematicreview’ definition to be UK-wide, European orinternational, and the nine-point criteria list wasexpanded or rephrased according to suggestions made(Appendix 8a). These replies were then collated andanalysed and a second feedback report was sent to thosewho had taken part (Appendix 8b).


Characteristics of included publications

Publications reported the methods or processes forconducting a systematic review or meta-analysis,described the methods for appraising a systematic reviewor meta-analysis, were reviews of reviews or were qualityor methodological evaluations. The majority of thepublications (76%) were published journal articles andwere published in English (93). Thirteen journal articleswere published in languages other than English. Theremaining publications were published reports, books,book chapters, Internet website publications, a trainingmanual and publications currently in press (Table 2).Publications focused primarily on the healthcare literature,although a limited number of papers addressed otheracademic fields such as education, criminology,management knowledge and public and social policy.

Sixty per cent of the publications were process or methodguidelines describing a methodological framework forconducting a systematic review or meta-analysis, ordiscussed the essential quality criteria that determined asystematic review and differentiated it from anotherreview type. These publications ranged from exhaustiveand comprehensive methodological guidelines to morerudimentary, introductory one- or two-page articles.

These briefer publications reported the basic principles ofsystematic reviews without providing step-by-stepguidance.

Thirty-six of the papers were studies and 27 werepublications describing how to evaluate or criticallyappraise a systematic review or meta-analysis. Eighteen ofthe publications were reviews of reviews. Of the 36studies, 28 were quality evaluations that analysed themethodological and reporting aspects of review articlespublished in peer-reviewed and non peer-reviewedjournals, compared the quality of Cochrane reviews andsystematic reviews published in paper-based journals, andcompared quality checklists for critically appraising thereport of a systematic review.

Data analysis

The frequency of the main quality criteria was as follows.

• Reporting a search strategy for identifying potentiallyrelevant literature was the review criteria item mostfrequently cited by authors. Ninety-seven per cent ofall publications stated that reviewers should verify thatthey had conducted a systematic literature search.


Results – literature review

Publication type Total

Published journal article 145Report 20Book 5Book chapter 15Internet website publication 2Unpublished 2

‘In press’ journal article 1‘In press’ report 1

Description Total

Process guideline 114Evaluation study 33Survey 2Prospective observational study 1Workshop-based study 1Paper describing how to critically 27

appraise a systematic reviewReview of reviews 18

Language Total

English 178Foreign language 13

Spanish 3German 3Italian 1French 3Dutch 3

Table 2: Study characteristics

• Ninety per cent stated that the synthesis of data tointegrate the results of all included studies was thenext most important quality criterion. This was nearlymatched by the need for a focused question (90%),the assessment of study quality (89%) and theinclusion of pre-determined written criteria for theselection of studies (89%).

• The methods for extracting data (74%) and selectingstudies for inclusion (73%) were viewed to be the nextmost significant criteria when conducting a systematicreview.

• Forty-nine per cent of publications reported the needfor a pre-defined study protocol and less than half(34%) described the structure of the final report.

• A report of the rationale for undertaking a systematicreview was considered by only 23 publications to be anessential quality component (13%).

See Table 3 for frequency of cited quality criteria.

Frequency ratings were also compiled for constituentparts within each criterion (see also Table 4, p12).

• Within the search strategy criterion, most publications(85%) cited the reporting of the methods of theelectronic database search for the identification andretrieval of potentially relevant studies.

• Reporting of a search for grey/unpublished literature(59%), the undertaking of hand searches (55%),contact with experts in the field (56%), and thescanning of reference lists of books and relevantarticles (54%) were all cited with a similar frequency.

• Whereas 50% of publications stated that languagerestrictions should be reported, only 26% expressedthat the years searched should also be reported.

• Only 22 (11%) publications stated that a search of theInternet should be conducted.

Discussion about the synthesising of data differed.Seventy-three per cent of publications discussed thesynthesis of data from primary studies within the contextof meta-analysis and 38% also presented cases wherethis could be accomplished qualitatively. Clearly, a meta-analysis or analysis of pooled data could not be a criterionas a quantitative synthesis of the results is not alwayspossible owing to the heterogeneity of included studies,along with other reasons, such as lack of data. Forpublications that discussed the process of meta-analysis,47% confirmed that the methods of that analysis shouldbe clearly reported. Sixty per cent agreed that a test forheterogeneity should be reported, while less than half(42%) stated that an examination of publication bias anda sensitivity analysis (36%) should be conducted.

Although 87% of authors reported that the validity ofincluded studies must be assessed, only 50% certified the need for the use of a validated quality appraisalinstrument or checklist. Forty-five per cent stated that thenumber of reviewers appraising publications should bereported and whether or not this had been done blinded;22% of papers cited the need to indicate whether or notreviewers had worked independently. Less than half ofpublications concurred that for reviews of interventionsthe inclusion and exclusion criteria should be defined bypopulation, type of intervention, outcome or studydesign.

Half of publications cited the need to describe the levelsof screening and the application of inclusion andexclusion criteria (50%) during the exclusion of papers.Forty-two per cent maintained that the number ofreviewers excluding papers should be reported. Less thana quarter (12%) of authors cited the need to reportwhether or not inter-rater agreement by kappa statisticshould be assessed to qualify agreement betweenreviewers.

Just over a quarter (31%) of papers agreed that the useof a data extraction form should be reported. Thirty-sixper cent cited the need to report the number of reviewersextracting data during data extraction. Masking ofreviewers, whether or not they worked independentlyand if and how they had resolved disagreement was citedless frequently (18% and 24%, respectively). Thirty-onepublications (16%) affirmed that any attempt to obtainunpublished or missing data should be reported.


Quality criteria Frequency (%)

Search strategy 186 97%Data synthesis 172 90%Focused question 165 90%Study inclusion/exclusion criteria 167 87%Study quality assessment 168 87%Data extraction 143 74%Study selection 137 71%Protocol 94 49%Reporting and dissemination 64 33%Rationale 23 13%

Table 3: Frequency of cited quality criteria

Quality criteria and components Frequency (%)

Search strategy 186 97%Electronic databases 163 85%List of databases to be searched 120 63%Grey literature 114 59%Contact with experts 108 56%Search terms to be used 104 54%Hand searching 106 55%Reference lists 103 54%Design of the search strategy 95 50%Specified language constraints 95 50%Internet 22 11%Years searched 51 26%Update search 9 5%

Data synthesis 172 90%Quantitative data/meta-analysis 140 73%Heterogeneity/homogeneity 114 60%Statistical model/methods used 91 47%Publication bias 80 42%Descriptive data synthesis 70 36%Sensitivity analysis 70 36%Effect measure 66 34%Outcome measures 52 27%Comparisons 23 12%

Study quality assessment 168 87%Quality appraisal instruments 96 50%Number and masking of reviewers 87 45%Between-reviewer disagreement 67 34%Working independently 42 22%Design and components 26 14%Development 23 13%Pilot 16 8%

Study inclusion/exclusion criteria 167 87%Outcomes to be considered 91 47%Type(s) of intervention considered (and comparison intervention) 90 47%Types of study design 88 46%Population 86 45%Problem of interest 42 22%

Data extraction 143 74%Development 17 9%Number and masking of reviewers 69 36%Data extraction form 60 31%


Table 4: Frequency of cited quality criteria components*

* For the organisation of these criteria in the data extraction see Appendix 4, p35.


Quality criteria and components Frequency (%)

Between-reviewer disagreement 47 24%Working independently 35 18%Unpublished/missing data 31 16%Pilot 23 12%Design and content 22 11%Comparison of data 23 12%

Study selection 137 71%Application of inclusion/exclusion criteria 95 50%Levels of screening 96 50%Number of reviewers 80 42%Included/excluded papers 37 19%Kappa score 23 12%

Protocol 94 49%Focused main question (and any secondary questions) 49 25%Search strategy 42 22%Study inclusion/exclusion criteria (study design, types of participant) 41 21%Data analysis strategy 33 17%Data extraction and synthesis of extracted data 33 17%Steering/review group 26 14%Study quality assessment and procedures 27 14%Study selection criteria and procedures 17 9%Outcome measures to be assessed 15 8%Intervention/exposure of interest 14 8%

Reporting and dissemination 12 6%Reporting and dissemination 64 33%Results of the review 42 22%Conclusions 34 18%Review methods 35 18%Details of the included/excluded studies 31 16%Discussion 30 16%Review question(s) 29 15%Background information 26 14%Executive summary/abstract 24 13%Dissemination activities 21 11%Review limitations 20 11%Peer review 19 10%References and appendices 17 9%Review update 13 7%

Table 4: Frequency of cited quality criteria components (cont.)

The first Delphi survey was sent out by email to 27organisations and received 20 replies with 16 fullycompleted questionnaires. The results can be summarisedas follows (see also Table 5).

Thirteen out of 16 agreed that there should be a UK-wideconsensus for the definition of a systematic review,although three were strongly against this and gave goodreasons for their objection (see p48). The nine proposedquality criteria were approved unanimously (four criteria)

or by a great majority (14 or 15 out of 16). The lowestlevel of agreement was 13 out of 16 for the inclusion ofa protocol.

Based on these findings, the second Delphi survey wassent out to the 16 who had replied to stage 1 andreceived 11 replies with 10 completed questionnaires.The majority (9 out of 10) supported the view that adefinition of ‘a systematic review’ should be agreed foruse in the UK but should also meet the definitions used


Delphi survey results

Yes No

Question 1: Is it desirable to seek a UK-wide consensus 13 3for the definition of a systematic review?Comments: 8 3

Question 2: Should the following quality criteria be included when reporting a systematic review?2.1 Protocol 13 32.2 Focused question 15 12.3 Search strategy 162.4 Years searched 162.5 Study inclusion/exclusion criteria 162.6 Study selection 162.7 Study quality assessment 15 12.8 Data extraction 14+1 query 12.9 Data synthesis 14+1 query 12.10 Are there any further criteria that you think should be 9 2

included in a report of a systematic review?Comments: 13 2

Question 3: Is it desirable to seek a UK-wide consensus on criteria for the definition and quality standards for:3.1 A literature review 5 113.2 A consensus group statement 10 63.3 A review of systematic reviews 10 7*Comments: 7 3

Table 5: Summary of first round Delphi survey results

*One answered ‘yes and no’ to point 3.3.

worldwide by the Cochrane and CampbellCollaborations. Only one preferred a definition to beagreed for use within the UK only.

The list of nine quality criteria was extended by a furtherfive suggested by the first-stage replies, and a rewordedversion was offered for all criteria that had not beenunanimously approved during the first round. Preferenceswere evenly matched between those who preferred theoriginal wording and those who favoured the newversion. The majority approved of the additional criteria (8 out of 10) with two being against the idea of astandardised list of potential biases.

Numbers in Table 6 are matched to those in question 2of the first Delphi survey. The first four criteria – searchstrategy, years searched, study inclusion/exclusion criteriaand study selection – are those which receivedunanimous approval from the first round. Those who

marked them the second time still agreed with this. Noteveryone replied to the listings in question 2 – there ispotentially a maximum of eight replies to each point.

Full details of the results analysis (including all commentsfrom those who responded) are given in Appendix 7b(first round Delphi survey) and Appendix 8b (secondround Delphi survey).

The Delphi survey results suggest that the most frequentlycited quality criteria (see Table 3) do indeed provide arealistic listing to be used for reporting systematicreviews, but those who are prepared to include additionalcriteria could usefully include those added by this Delphiconsensus exercise.


Question 1: Should a UK definition of ‘a systematic review’ be: Rating Rating Rating1 2 3

1a Agreed for use in the UK, in UK health and social services 1 2 7and other areas of UK public policy

1b Agreed for use in the UK and across Europe 7 31c Agreed for use in the UK, and meeting the definitions used 9 1

worldwide by the Cochrane and Campbell Collaborations

Yes NoQuestion 2: Should the following quality criteria be included when reporting a systematic review?2.3 Search strategy2.4 Years searched2.5 Study inclusion/exclusion criteria2.6 Study selection2.1 Protocol 22.1a Published protocol 3 32.2 Focused question 32.2a Focused question(s) 52.7 Study quality assessment 32.7a Method(s) of study quality assessment 52.8 Data extraction 32.8a Method(s) of data extraction 4 12.9 Data synthesis 32.9a Method(s) of data synthesis 42.11 Funding source or sponsorship 82.12 Declaration of prior interests 72.13 Standardised list of potential biases 6 22.14 Table of included studies 82.15 Statistical or other assessment of the heterogeneity 8

of included studies

Table 6: Summary of second round Delphi survey results

In conducting this literature review, robust supportingconsensus evidence was identified for seven of the tenquality criteria. Five of the criteria components were citedin 89% of publications, indicating that the majority of theauthors viewed these components as key criteria. Forexample, almost all of the authors stated that a reviewshould systematically verify that a comprehensiveliterature search was conducted and that the methods ofthe search strategy are reported. Two of the other criteriacomponents were cited in almost 75% of the papers,once again showing some consistency of agreement.Although the three remaining criteria were cited lessfrequently, one of these, the ‘protocol’ criterion, wascited in almost half of publications.

These findings are consistent with current standardcriteria used by some UK organisations. For example thefour criteria used by the Centre for Reviews andDissemination for the Database of Reviews of Effects(DARE) are included in the top five quality criteria

components most frequently cited in this review (Centre for Reviews and Dissemination 2001). Similarly,the Centre’s process guidance incorporates all sevencriteria rated above 70% by publications in this review(Centre for Reviews and Dissemination 2000). TheCochrane Handbook for Systematic Reviews ofInterventions also includes all of the seven quality criteriarated above 70% (Green and Higgins 2004).

In addition, the criteria used by Oxman and Guyatt (1991)to assess the validity of an index of the scientific qualityof research overviews are contained in these sevencriteria, with the exception of the criterion ‘focusedquestion’.

The ‘priority’ criteria identified in this review are alsoconsistent with ‘gold standard’ internationaldevelopments to obtain consensus agreement formethodological guidelines and reporting standards foroverviews (Cook et al. 1995; Moher et al. 1999; Stroup


Discussion

Criteria rated >70%

‘Gold standard’ Search Data Focused Study Study Data Study %criteria strategy synthesis question inclusion/ quality extraction selection

exclusion assessmentcriteria

Centre for Reviews ! ! ! ! ! ! ! 100%and DisseminationCochrane Collaboration ! ! ! ! ! ! ! 100%Oxman and Guyatt 1991 ! ! " ! ! ! ! 86%MOOSE ! ! ! ! ! ! ! 100%Potsdam Consultation ! ! ! ! ! ! ! 100%QUOROM ! ! ! ! ! ! ! 100%AHRQ (Systems to Rate ! ! ! ! ! ! ! 100%the Strength of Evidence)

Table 7: Comparison of quality criteria rated >70% to ‘gold standard’ criteria


et al. 2000; West et al. 2002). This finding is notunexpected as there was some overlap betweenpublications included in this review and some of the UKorganisations that provide what is currently viewed asstandard recommended guidance for conductingsystematic reviews. Some consistency would, therefore,have been expected between the most frequently citedcriteria components identified in the review and theguidance produced by key UK organisations (Table 7).

Additionally, a total of 74 publications (38%) eitherreferred to or based their process guidance directly onthat developed by one of the key organisations (Table 8).Fifty-two (27%) of publications either cited or suggestedthat reviewers use the Oxman and Guyatt (1991)checklist. Thirty-one of these publications were evaluationstudies, and of these, 21 (67%) used the checklist. This suggests that a high percentage of currentmethodological evaluation studies endorse the use of theOxman and Guyatt checklist.

Findings for those criteria components that had beencited less frequently by authors should be interpretedwith some caution. It would be misleading to assumethat a less frequently cited criteria component means thatthe item is not an important component of the reviewprocess. For example, only 26% of reviewers agreed thatthe ‘years searched’ during the electronic database searchshould be reported. Identifying the years searched allowsfor transparency and reproducibility of results and permitsa reader to assess the comprehensiveness of the searchstrategy. In addition, Quality of Reporting of Meta-analyses (QUOROM) reporting guidelines suggest that‘years considered’ should be discussed when reportingthe literature search methods, as QUOROM participantsagreed that this factor was an important feature in achecklist of reporting standards (Moher et al. 1999).

Another example is the ‘review protocol’ criterion. Only 48% of publications agreed that reviewers shouldprovide evidence of a review protocol where the reviewhypothesis and methods have been declared prior to

undertaking a review. Although this is less than 50%agreement, it should not be interpreted to mean that thedevelopment of a review protocol, in advance of thereview, is an inconsiderable feature of the reviewmethod. It may simply reflect that authors do notperceive the review protocol as a separate item from themain review.

A recent retrospective comparative study found that outof 47 new Cochrane reviews, 43 had at least one sectionthat had undergone a major change compared with itsmost recently published protocol and that the greatestvariation between protocols and reviews was in themethods section (Silagy et al. 2002). Preparation of areview protocol prior to the review minimises bias bydeclaring the methods at the start. Clearly, Silagy et al.(2002) show that changes made to the protocol thatcould have been based on prior knowledge of the resultsmight have introduced the possibility of bias.

This review has attempted to determine the qualitycriteria that reviewers collectively consider to be theprimary measures that define a systematic review, and toestablish whether it would be desirable to seek a UKconsensus for the definition of a systematic review. Asthe results of the Delphi survey showed that 81% ofthose organisations surveyed agreed that such aconsensus would be desirable, it may now be anappropriate time to move forward with developing adefinition.

A consensus agreement would enhance the overallquality of systematic reviews and the review process, andprovide reviewers, readers and database indexers with anagreed definition. The Delphi survey results suggest thatthe most frequently cited quality criteria (see Table 3) doindeed provide a realistic listing to be used for reportingsystematic reviews, but those who are prepared toinclude additional criteria could usefully incorporate thefurther items suggested.

Criteria Frequency %

Oxman and Guyatt 1991 52 27%Centre for Reviews and Dissemination 11 6%Cochrane Collaboration 8 4%QUOROM 3 2%

Table 8: Frequency of publications using existing criteria

Limitations of this review

A number of limitations can be identified in this review in addition to those limitations inherent in the reviewmethods themselves.

This review was conducted by a single reviewer and keydata may have been missed or misinterpreted during dataextraction (and included publications were not alwaysexplicit in their guidance). If guidance did not clearlyanswer or consider a quality criteria question, it was notscored on the data abstraction sheet. Steps in the reviewprocess were reported with varying strength andemphasis and this may have resulted in an item receivinga score from one reviewer that might have not beenselected by another (Carr 2002; Koretz 2002). As therewas no second reviewer this could not be confirmed orqualified by inter-rater agreement. In addition, it isnecessary to consider that criteria discussed by reviewersmay have been selected by subjective judgement.

Random errors may have occurred during data analysis,as scores were calculated manually. Although totalfrequency scores were calculated more than once foreach criterion question and for all criteria components theoverall reliability was not examined. At best, frequencyscores must be viewed as estimated.

As this review was conducted within a restrictedtimeframe, an updated electronic database search wasnot carried out. Relevant papers published after the endof the search date will, therefore, have been missed.

Conclusions

The results of the literature review show that there is someagreement among reviewers on the ‘primary’ qualitycriteria components for a systematic review and that thereis an emerging hierarchical structure where reviewers agreeon the principal elements defining a systematic review. Theresults of the Delphi survey further confirm agreement ofthe highlighted criteria. In part this was to be expectedsince some, though not all, of those organisationsparticipating in the survey would have been involved inproducing review guidelines. The detailed comments in thesurvey responses serve to reinforce the rationale behindtheir approval. It is this agreement or consensus that mayultimately determine an established and approveddefinition of a systematic review for application in the UK.

Although the systematic review process is an evolvingone, the overall results of this review suggest that mostauthors currently undertaking systematic reviews agreeon the fundamental or ‘core quality criteria’ thatdistinguish a systematic review from other review types.


Text references*

Carr AB (2002) Systematic review of the literature: theoverview and meta-analysis. Dental Clinics of NorthAmerica 46 (1):79–86.

Centre for Reviews and Dissemination (2000) Manual for Selecting Reviews and Writing Abstracts for the Database of Abstracts of Reviews of Effects (DARE).York: Centre for Reviews and Dissemination. (Not a public document – contact CRD for information)

Centre for Reviews and Dissemination (2001)Undertaking Systematic Reviews of Research onEffectiveness. CRD’s Guidance for those Carrying out orCommissioning Reviews. CRD Report 4. 2nd edition.York: Centre for Reviews and Dissemination.

Cook DJ, Guyatt GH, Laupacis A et al. (1992) Rules ofevidence and clinical recommendations on the use ofantithrombotic agents. Chest 102 (Suppl 4):305S–311S.

Cook DJ, Sackett DL, Spitzer W (1995) Methodologicguidelines for systematic reviews of randomisedcontrolled trials in health care from the Potsdamconsultation on meta-analysis. Journal of ClinicalEpidemiology 48 (1):167–71.

Delbecq AL, van den Venn AH, Gustafson DH (1975)Group Techniques for Program Planning: a Guide toNominal Group and Delphi Processes. New York: Scott,Foresman.

Ernst E, Pittler MH (2001) Assessment of therapeuticsafety in systematic reviews: literature review. BritishMedical Journal 323 (7312):546.

Forbes DA (2003) An example of the use of systematicreviews to answer an effectiveness question. WesternJournal of Nursing Research 25 (2):179–92.

Gray JAM (1997) Evidence-based Healthcare: How toMake Health Policy and Management Decisions.Edinburgh: Churchill Livingstone.

Green S, Higgins J, editors (2004) Cochrane Handbookfor Systematic Reviews of Interventions 4.2.2. In: The Cochrane Library, Issue 1, 2004. Chichester:Wiley. Updated quarterly.

Grootens KP, Assendelft WJ, Overbeke AJ (2003)Increased number of systematic reviews in theNetherlands in the period 1991–2000. NederlandsTijdschrift voor Geneeskunde 147 (45):2226–30.

Hemels ME, Vicente C, Sadri H et al. (2004) Qualityassessment of meta-analyses of RCTs of pharmacotherapyin major depressive disorder. Current Medical Researchand Opinion 20 (4):477–84.

Jackson N, Waters E (in draft) Guidelines for systematicreviews of health promotions and public healthinterventions. Victoria, Australia: Cochrane HealthPromotion and Public Health Field.

Jadad AR, Moher M, Browman P et al. (2000) Systematicreviews and meta-analyses on treatment of asthma:critical evaluation. British Medical Journal 320:537–40.

Koretz RL (2002) Methods of meta-analysis: an analysis.Current Opinion in Clinical Nutrition and Metabolic Care5:467–74.


References

* Note some publications listed here are included in the analysis.See p21 for list of included publications.


Macdonald G (2003) Using Systematic Reviews to ImproveSocial Care. SCIE Reports: No 4. London: Social Care Institute for Excellence.

McBride AJ, Pates R, Ramadan R et al. (2003) Delphisurvey of experts’ opinions on strategies used bycommunity pharmacists to reduce over-the-counter drugmisuse. Addiction 98:487–97.

Moher D, Cook DJ, Eastwood S et al. (1999) Improvingthe quality of reports of meta-analyses of randomisedcontrolled trials: the QUOROM statement. Lancet 354(9193):1896–900.

Oakley A (2002) Social science and evidence-basedeverything: the case of education. Educational Review 54(3):277–86.

Oxman AD, Guyatt GH (1991) Validation of an index ofthe quality of review articles. Journal of ClinicalEpidemiology 44 (11):1271–8.

Oxman AD, Guyatt GH, Singer J et al. (1991) Agreementamong reviewers of review articles. Journal of ClinicalEpidemiology 44 (1):91–8.

Petrosino A, Boruch RF, Farrington DP et al. (2003)Toward evidence-based criminology and criminal justice:systematic reviews, the Campbell collaboration, and theCrime and Justice Group. International Journal ofComparative Criminology 3 (1):42–61.

Petticrew M (2005) Systematic Reviews in the SocialSciences: A Practical Handbook. Abingdon: BlackwellPublishing.

Rowe G, Wright, G. (1999) The Delphi technique as aforecasting tool: issues and analysis. International Journalof Forecasting 15:353–75.

Shea B, Dube C, Moher D (2001) Assessing the quality ofreports of systematic reviews: the QUOROM statementcompared to other tools. In: Chalmers I, Egger M, SmithGD et al., editors. Systematic Reviews in Health Care:Meta-analysis in Context. 2nd edition. London: BMJBooks.

Shea B, Moher M, Graham I et al. (2002) A comparison ofthe quality of Cochrane reviews and systematic reviewspublished in paper-based journals. Evaluation and theHealth Professions 25 (1):116–29.

Silagy CA, Middleton P, Hopewell S et al. (2002)Publishing protocols of systematic reviews: comparingwhat was done to what was planned. Journal of theAmerican Medical Association 287 (21):2831–4.

Stroup DF, Berlin JA, Morton SC et al. (2000) Meta-analysis of observational studies in epidemiology: aproposal for reporting. Journal of the American MedicalAssociation 283 (15):2008–12.

Verhagen AP, de Vet HCW, de Bie RA et al. (1998) TheDelphi List: a criteria list for quality assessment ofrandomized clinical trials for constructing systematicreviews developed by Delphi consensus. Journal of ClinicalEpidemiology 51 (12):1235–41.

West S, King V, Carey TS et al. (2002) Systems to Ratethe Strength of Scientific Evidence. EvidenceReport/Technology Assessment Number 47. Rockville:Agency for Healthcare Research and Quality.

Included publications

Anderson CM, Overend TJ, Lucy SD (2002) Changingpractice through a systematic review: reflections fromexperience. Physiotherapy Canada 54 (3):186–98.

Anderson LM, Fielding JE, Fullilove MT et al. (2003)Methods for conducting systematic reviews of theevidence of effectiveness and economic efficiency ofinterventions to promote healthy social environments.American Journal of Preventive Medicine 24 (3):25–31.

Arrieta JML, Qizilbash N (1996) Evidence-based medicine:systematic reviews. The Cochrane Collaboration.Medicina Clinica 107 (15):581–5.

Assendelft WJ, Koes BW, Knipschild PG et al. (1995) Therelationship between methodological quality andconclusions in reviews of spinal manipulation. Journal ofthe American Medical Association 274 (24):1942–8.

Assendelft WJJ, Scholten RJPM, Hoving JL et al. (2001)Systematic reviews in practice. VIII. Searching andassessing systematic reviews. Nederlands Tijdschrift voorGeneeskunde 145 (34):1625–31.

Auperin A, Pignon J-P, Poynard T. (1997) Review article:critical review of meta-analyses of randomized clinicaltrails in hepatogastroenterology. AlimentaryPharmacology and Therapeutics 11:215–25.

Bader J, Ismail A (2004) Survey of systematic reviews indentistry. Journal of the American Dental Association135 (4):464–73.

Bannigan K, Droogan J, Entwistle V (1997) Systematicreviews: what do they involve? Nursing Times 93 (18):52–3.

Barnes DE, Bero LA (1998) Why review articles on thehealth effects of passive smoking reach differentconclusions. Journal of the American Academy ofDermatology 279 (19):1566–70.

Beard MT, Bailey CB, Marsh V (2001) Meta-analysis: asystematic approach to scientific inquiry. Journal ofTheory Construction and Testing 5 (2):63–5.

Berman NG, Parker RA (2002) Meta-analysis: neitherquick nor easy. BMC Medical Research Methodology2 (10):1–9.

Bero LA, Grilli R, Grimshaw M et al. (1998) Closing thegap between research and practice: an overview ofsystematic reviews of interventions to promote theimplementation of research findings. British MedicalJournal 317 (7156):465–8.

Bhandari M, Morrow F, Kulkarni AV et al. (2001) Meta-analyses in orthopaedic surgery. A systematic review oftheir methodologies. Journal of Bone and Joint Surgery(American) 83 (1):15–24.

Bhandari M, Guyatt GH, Montori V et al. (2002) User'sguide to the orthopaedic literature: how to use asystematic literature review. Journal of Bone and JointSurgery (American) 84 (9):1672–82.

Bhandari M, Devereaux PJ, Montori V et al. (2004) Users'guide to the surgical literature: how to use a systematicliterature review and meta-analysis. Canadian Journal ofSurgery 47 (1):60–7.

Bhandari M, Montori VM, Devereaux PJ et al. (2004)Doubling the impact: publication of systematic reviewarticles in orthopaedic journals. Journal of Bone and JointSurgery (American) 86 (5):1012–6.

Bigby M, Williams H (2003) Appraising systematic reviewsand meta-analyses. Archives of Dermatology 139(6):795–8.

Bjordal JM, Greve G (1998) What may alter theconclusions of reviews? Physical Therapy Reviews3:121–32.

Blair A, Burg J, Foran J et al. (1995) Guidelines forapplication of meta-analysis in environmentalepidemiology. Regulatory Toxicology and Pharmacology22 (2):189–97.

Bramwell VH, Williams CJ (1997) Do authors of reviewarticles use systematic methods to identify, assess andsynthesize information? Annals of Oncology 8:1185–95.

Brealey S, Glenny A (1999) A framework forradiographers planning to undertake a systematic review.Radiography 5 (3):131–46.

Breslow RA, Ros SA, Weed DL (1998) Quality of reviewsin epidemiology. American Journal of Public Health 88(3):475–7.


Bull J, Mulvihill C, Quigley R (2003) Prevention of lowbirth weight: assessing the effectiveness of smokingcessation and nutritional interventions. Evidence Briefing.London: Health Development Agency.

Busse R, Orvain J, Velasco M et al. (2002) Best practice inundertaking and reporting health technology assessments.Working group 4 report. International Journal ofTechnology Assessment in Health Care 18 (2):361–422.

Campbell Collaboration (2001) Campbell systematicreviews: guidelines for the preparation of reviewprotocols. www.campbellcollaboration.org [accessed 8August 2005].

Campbell MK, Daly C, Wallace SA et al. (2000) Evidence-based medicine in nephrology: identifying and criticallyappraising the literature. Nephrology DialysisTransplantation 15:1950–5.

Canning U, Millward L, Raj T et al. (2004) Drug useprevention among young people: a review of reviews.Evidence Briefing. London: Health Development Agency.

Carr AB (2002) Systematic review of the literature: theoverview and meta-analysis. Dental Clinics of NorthAmerica 46 (1):79–86.

Centre for Reviews and Dissemination (1999) Gettingevidence into practice. Effective Health Care 5 (1):1–16.

Centre for Reviews and Dissemination (2000) Manual for selecting reviews and writing abstracts for the Database of Abstracts of Reviews of Effects (DARE).York: Centre for Reviews and Dissemination. (Not a public document – contact CRD for information)

Centre for Reviews and Dissemination (2001)Undertaking Systematic Reviews of Research onEffectiveness. CRD’s Guidance for those Carrying out orCommissioning Reviews. CRD Report 4. 2nd edition.York: Centre for Reviews and Dissemination.

Choi PT, Halpern SH, Malik N et al. (2001) Examining theevidence in anesthesia literature: a critical appraisal ofsystematic reviews. Anesthesia and Analgesia 92 (3):700–9.

Christensen E. (2001) Quality of reporting of meta-analyses: the QUOROM statement. Will it help? Journal of Hepatology 34 (2):342–5.

Ciliska D, Cullum N, Marks S (2001) Evaluation ofsystematic reviews of treatment or preventioninterventions. EBN users' guide. Evidence Based Nursing4 (4):100–4.

Cook DJ, Sackett DL, Spitzer W (1995) Methodologicguidelines for systematic reviews of randomisedcontrolled trials in health care from the Potsdamconsultation on meta-analysis. Journal of ClinicalEpidemiology 48 (1):167–71.

Counsell C (1997) Formulating questions and locatingprimary studies for inclusion in systematic reviews. Annals of Internal Medicine 127 (5):380–7.

Cranney A, Tugwell P, Wells G et al. (2002) Systematicreviews of randomized trials in osteoporosis: Introductionand methodology. Endocrine Reviews 23 (4):497–507.

Crombie IK (1996) Appraising review papers. In: CrombieIK, editor. The Pocket Guide To Critical Appraisal: AHandbook For Health Care Professionals. London: BritishMedical Journal Publishing Group, p56–61.

Cullum N (1999) Critical appraisal. How to decide ifreview articles are trustworthy and relevant for practice.Nursing Times Learning Curve 3 (6):4–6.

D’Agostino RB, Weintraub M (1995) Meta-analysis: amethod for synthesizing research. Clinical Pharmacologyand Therapeutics 58 (6):605–16.

Darnand K, Fouque D (2000) How to produce systematicreviews of great quality and define evidenced basednephrology. Nephrologie 21 (2):71–4.

Davies HTO, Crombie IK (1998) Getting to grips withsystematic reviews and meta-analyses. Hospital Medicine59 (12):955–8.

Davies P (2004) Systematic reviews and the CampbellCollaboration. In: Thomas G, Pring R, editors. Evidence-based Practice in Education. Maidenhead: Open UniversityPress, p21–33.

de Bie RA (1996) Methodology of systematic reviews: anintroduction. Physical Therapy Review 1 (1):47–51.


de Vet HCW, de Bie RA, van der Heijden GJM et al.(1997) Systematic reviews on the basis of methodologicalcriteria. Physiotherapy 83 (6):284–9.

de Vet HCW, Verhagen AP, Logghe I (2002) Literatureresearch: goals and design of systematic reviews.Nederlands Tijdschrift voor Fysiotherapie 112 (3):80–5.

Deeks JJ (1998) Systematic reviews of published evidence:miracles or minefields? Annals of Oncology 9 (7):703–9.

Del Mar C, Glasziou P (2001) Studying systematic reviews.Journal of Family Practice 50 (11):955–7.

Deville WL, Buntinx F, Bouter LM et al. (2002) Conducting systematic reviews of diagnostic studies:didactic guidelines. BMC Medical Research Methodology2 (9):1–13.

Dickson R. Entwistle V (1996) Systematic reviews: keepingup with research evidence. Nursing Standard 10 (19):32.

Droogan J, Song F (1996) The process and importance ofsystematic reviews. Nurse Researcher 4 (1):15–26.

Droogan J, Cullum N (1998) Systematic reviews innursing. International Journal of Nursing Studies35 (1–2):13–22.

Durlak JA, Meerson I, Ewell Foster CJ (2003) Meta-analysis. In: Thomas JC, Hersen M, editors. UnderstandingResearch in Clinical and Counseling Psychology. London: Lawrence Erlbaum Associates, p243–67.

Egger M, Smith GD, Phillips AN (1997) Meta-analysis:principles and procedures. British Medical Journal 315(7121):1533–7.

Egger M, Davey Smith G, Altman DG (2001) Principles ofand procedures for systematic reviews. In: Egger M,Davey Smith G, Altman DG, editors. Systematic Reviewsin Health Care: meta-analysis in context. 2nd edition.London: BMJ Books, p23-42.

Ellis S, Barnett-Page E, Morgan A et al. (2003) HIV prevention: a review of reviews assessing theeffectiveness of interventions to reduce the risk of sexualtransmission. Evidence briefing. London: HealthDevelopment Agency.

Ellis S, Grey A (2004) Prevention of sexually transmittedinfections (STIs): a review of reviews into the effectivenessof non-clinical interventions. Evidence Briefing. London:Health Development Agency.

Evans D, Kowanko I, Hodgkinson B (1998) Systematicreviews in nursing research. Australian Nursing Journal5 (10):42.

Evans D, Chang A (2000) Appraising systematic reviews.Changing Practice Supplement. Adelaide: Joanna BriggsInstitute for Evidence Based Nursing and Midwifery.

Evans D, Kowanko I (2000) Literature reviews: evolutionof a research methodology. Australian Journal ofAdvanced Nursing 18 (2):33–8.

Evans D (2001) Systematic reviews of nursing research.Intensive and Critical Care Nursing 17 (1):51–7.

Evans J, Benefield P (2001) Systematic reviews ofeducational research: does the medical model fit? British Educational Research Journal 27 (5):527–41.

Farrington DP, Petrosino A, Welsh BC (2001) Systematicreviews and cost-benefit analyses of correctionalinterventions. Prison Journal 81 (3):339–59.

Fletcher J, Stewart-Brown S, Barlow J (1997) Systematicreview of reviews of the effectiveness of school-basedhealth promotion. Oxford: Health Services Research Unit,University of Oxford.

Forbes DA (2003) An example of the use of systematicreviews to answer an effectiveness question. WesternJournal of Nursing Research 25 (2):179–92.

Forward L, Hobby L (2002) A practical guide toconducting a systematic review. Nursing Times98 (2):36–7.

French S (2002) A user's guide to the chiropractic andosteopathic literature: how to read a systematic review.Australasian Chiropractic and Osteopathy 10 (2):87–90.

Furlan AD, Clarke J, Esmail R et al. (2001) A critical reviewof reviews on the treatment of chronic low back pain.Spine 26 (7):E155–62.


Gallagher EJ (1999) Systematic reviews: a logicalmethodological extension of evidence-based medicine.Academic Emergency Medicine 6 (12):1255–60.

Garcia JM, Rodriguez Martin JL, Subirana M (2003)Systematic reviews of the scientific evidence: somepractical points. Enfermería Clínica 13 (3):159–63.

Geddes J (2002) Systematic reviews and meta-analyses.In: Priebe S, Slade M, editors. Evidence in Mental HealthCare. Hove: Brunner-Routledge, p72-80.

Glasziou P, Irwig L, Bain C et al. (2001) SystematicReviews in Health Care. Cambridge: Cambridge UniversityPress.

Glenny AM, Worthington HV, Esposito M et al. (2003).The assessment of systematic reviews in dentistry. Journal of Dental Research 82:587.

Goodwin DM, Higginson IJ, Edwards AG et al. (2002) Anevaluation of systematic reviews of palliative care services.Journal of Palliative Care 18 (2):77–83.

Green S, Higgins J, editors (2004) Cochrane Handbookfor Systematic Reviews of Interventions 4.2.2. In: TheCochrane Library, Issue 1, 2004. Chichester: Wiley.Updated quarterly.

Greenhalgh T (1997) Papers that summarise other papers(systematic reviews and meta-analyses). In: Greenhalgh T,editor. How to Read a Paper: The Basics of EvidenceBased Medicine. London: BMJ Books, p111–27.

Greenhalgh T (1997) How to read a paper: papers thatsummarise other papers (systematic reviews and meta-analyses). British Medical Journal 315: 672–5.

Greenhalgh T, Robert G, Macfarlane F et al. Storylines ofresearch: a meta-narrative perspective on systematicreview. Social Science and Medicine. In press.

Grimshaw JM, Shirran L, Thomas R et al. (2001)Changing provider behavior an overview of systematic reviews of interventions. Medical Care 39 (8 Suppl 2):II2–45.

Grimshaw J, McAuley LM, Bero LA et al. (2003)Systematic reviews of the effectiveness of qualityimprovement strategies and programmes. Quality and Safety in Health Care 12 (4):298–303.

Guallar E, Damian J, Martin-Moreno JM (1997) Meta-analysis and systematic reviews in cardiology. Revista Espanola de Cardiologia 50 (5):345–54.

Guyatt G, Rennie D (2002) User’s guides to the medicalliterature. Chicago: American Medical Association Press.

Hailey D (2003) Toward transparency in healthtechnology assessment: a checklist for HTA reports.International Journal of Technology Assessment in HealthCare 19 (1):1–7.

Harden A (2001) The fine detail: conducting a systematicreview. In: Oliver S, Peersman G, editors. Using Researchfor Effective Health Promotion. Buckingham: OpenUniversity Press, p111–22.

Hearn J, Feuer D, Higginson IJ et al. (1999) Issues inresearch. Systematic reviews. Palliative Medicine13 (1):75–80.

Hense H-W, Antes G (1997) Systematic reviews incardiology. Concepts of evidence-based medicine and theCochrane Collaboration. Zeitschrift fur Kardiologie 86(5):313–9.

Hopayian K (2001) The need for caution in interpretinghigh quality systematic reviews. British Medical Journal323 (7314):681–4.

Horvath AR, Pewsner D (2004) Systematic reviews inlaboratory medicine: Principles, processes and practicalconsiderations. Clinica Chimica Acta 342 (1-2):23–39.

Hoving JL, Gross AR, Gasner D et al. (2001) A criticalappraisal of review articles on the effectiveness ofconservative treatment for neck pain. Spine 26(2):196–205.

Hughes EG (1996) Systematic literature review and meta-analysis. Seminars in Reproductive Endocrinology 14(2):161–9.


Hunt DL, McKibbon KA (1997) Locating and appraisingsystematic reviews. Annals of Internal Medicine 126(7):532–8.

Hunt DL, McKibbon KA (1998) Locating and appraisingsystematic reviews. In: Mulrow C, Cook D, editors.Systematic Reviews: Synthesis of Best Evidence for HealthCare Decisions. Philadelphia: American College ofPhysicians, p13–22.

Ismail AI, Bader JD, Kamerow DB (1999) Systematicreviews and the practice of evidence-based dentistry:professional and policy implications. Journal of theAmerican College of Dentists 66 (1):5–12.

Jackson SZ, Edwards RD, Kahan B et al. (2001) Anassessment of the methods and concepts used tosynthesize the evidence of effectiveness in healthpromotion: a review of 17 initiatives. Toronto: CanadianConsortium for Health Promotion Research.

Jadad AR, McQuay HJ (1996) Meta-analyses to evaluateanalgesic interventions: a systematic qualitative review oftheir methodology. Journal of Clinical Epidemiology49 (2):235–43.

Jadad AR, Cook DJ, Jones A et al. (1998) Methodologyand reports of systematic reviews and meta-analyses:comparison of Cochrane reviews with articles publishedin paper-based journals. Journal of the American MedicalAssociation 280 (3):278–80.

Jadad AR, Klassen TP, Moher D et al. (1998) Guides forreading and interpreting systematic reviews: I. Gettingstarted; II. How did the authors find the studies andassess their quality?; III. How did the authors synthesizethe data and make their conclusions? Archives ofPediatrics and Adolescent Medicine 152(7,8,9):700–4;812–7;915–20.

Jadad AR, Booker L, Gould M et al. (1999) The treatmentof attention-deficit hyperactivity disorder: an annotatedbibliography and critical appraisal of published systematicreviews and meta-analyses. Canadian Journal ofPsychiatry 44: 1025–35.

Jadad AR, Moher M, Browman P et al. (2000) Systematicreviews and meta-analyses on treatment of asthma:critical evaluation. British Medical Journal 320:537–40.

Jefferson T, Demicheli V, Vale L (2002) Quality ofsystematic reviews of economic evaluations in healthcare. Journal of the American Medical Association287 (21):2809–12.

Jenicek M (2003) Foundations of Evidence-basedMedicine. London: Parthenon Publishing Group.

Joanna Briggs Institute (2004) Comprehensive SystematicReview Training Program. Adelaide: Joanna BriggsInstitute.

Jones J, Nelson A (2001) Demystifying systematic reviews.Nursing Times 97 (48):61–2.

Jones T, Evans D (2000) Conducting a systematic review.Australian Critical Care 13 (2):66–71.

Kelly KD, Travers A, Dorgan M et al. (2001) Evaluatingthe quality of systematic reviews in the emergencymedicine literature. Annals of Emergency Medicine 38(5):518–26.

Khan KS, Dinnes J, Kleijnen J (2001) Systematic review toevaluate diagnostic tests. European Journal of Obstetricsand Reproductive Biology 95:6–11.

Khan KS, Kunz R, Kleijnen J et al. (2003 Five steps toconducting a systematic review. Journal of the RoyalSociety of Medicine 96 (3):11821.

Khan KS, Kunz R, Kleijnen J et al. (2003) SystematicReviews to Support Evidence-based Medicine. London: Royal Society of Medicine Press.

Koop PM, Burgess-Pinto E (2003) Conducting asystematic review of the literature: a brief guide.Canadian Oncology Nursing Journal 13 (1):5–7.

Koretz RL (2002) Methods of meta-analysis: an analysis.Current Opinion in Clinical Nutrition and Metabolic Care5:467–74.

Kranke P, Eberhart LHJ (2002) Systematic reviews inanaesthesia… methods, implications and interpretation.Anasthesiologie and Intensivmedizin 43 (7–8):399–410.


Liberati A, Sheldon TA, Banta HD, EUR-ASSESS ProjectSubgroup report on Methodology (1997) Methodologicalguidance for the conduct of health technologyassessment. International Journal of TechnologyAssessment in Health Care 13 (2):186–219.

Linde K, ter Riet G, Hondras M et al. (2003)Characteristics and quality of systematic reviews ofacupuncture, herbal medicines, and homeopathy.Forschende Komplementarmedizin und KlassischeNaturheilkunde 10 (2):88–94.

Linde K, Willich SN (2003) How objective are systematicreviews? Differences between reviews on complementarymedicine. Journal of the Royal Society of Medicine 96(1):17–22.

Lipp A (2003) A guide to developing a systematic review.AORN Journal 78 (1):90–4.

Lipp A (2003) Unmasking the evidence: a systematicreview. ACORN (Journal of Perioperative Nursing inAustralia) 16 (4):16–9.

Lohr KN (2004) Rating the strength of scientific evidence:relevance for quality improvement programs.International Journal for Quality in Health Care 16(1):9–18.

Macbeth F, Overgaard J (2002) Expert reviews, systematicreviews and meta-analyses. Radiotherapy and Oncology64 (3):233–4.

Mackay HC, Barkham M, Rees A et al. (2003) Appraisalof published reviews of research on psychotherapy andcounselling with adults 1990–1998. Journal of Consultingand Clinical Psychology 71 (4): 652–6.

Magarey JM (2001) Elements of a systematic review.International Journal of Nursing Practice 7:376–82.

Massy-Westropp N, Masters M (2003) Doing systematicreviews in an occupational therapy department. BritishJournal of Occupational Therapy 66 (9):427–30.

McAlister, Clark HD, van Walraven C et al. (1999) Themedical review article revisited: has the science improved?Annals of Internal Medicine 131 (12):947–51.

McKibbon A (1999) Evidence-based principles andpractice. Sidcup: Harcourt Publishers.

Moher D, Cook DJ, Eastwood S et al. (1999) Improvingthe quality of reports of meta-analyses of randomisedcontrolled trials: the QUOROM statement. Lancet 354(9193):1896–900.

Moher D, Soeken K, Sampson M et al. (2002) Assessingthe quality of reports of systematic reviews in pediatriccomplementary and alternative medicine. BMC Pediatrics2 (1):3.

Moncrieff J (1998) Research synthesis: systematic reviewsand meta-analysis. International Review of Psychiatry10 (4):304–11.

Montori VM, Swiontkowski MF, Cook DJ et al. (2003)Methodologic issues in systematic reviews and meta-analyses. Clinical Orthopaedics and Related Research413:43–54.

Moynihan R (2004) Evaluating Health Services: A ReporterCovers the Science of Research Synthesis. New York:Millbank Memorial Fund.

Mulrow C, Cook D, editors (1997) Systematic reviewseries: 1) Synthesis of best evidence for clinical decisions2) Formulating questions and locating primary studies forinclusion in systematic reviews 3) Selecting and appraisingstudies for a systematic review 4) Quantitative synthesis insystematic reviews. Annals of Internal Medicine 126,127(5,7,9):376–80;531–7;380–7;820–6.

National Institute for Clinical Excellence (2004) Guide tothe Methods of Technology Appraisal. Reference N0515.London: National Institute for Clinical Excellence.

Needleman IG (2002) A guide to systematic reviews.Journal of Clinical Periodontology 9 (Suppl 3):6–9.

Nony P, Boissel JP, Lievre M et al. (1995) Introduction tometa-analytic methodology. Revue de Medecine Interne16 (7):536–46.

Nony P, Cucherat M, Haugh MC et al. (1995) Criticalreading of the meta-analysis of clinical trials. Therapie50 (4):339–51


O’Mathuna DP (2000) Evidence-based practice andreviews of therapeutic touch. Journal of NursingScholarship 32 (3):279–85.

Oakley A (2003) Research evidence, knowledgemanagement and educational practice: early lessons froma systematic approach. London Review of Education 1(1):21–33.

Oosterhuis WP, Niessen RW, Bossuyt PM (2000) Thescience of systematic reviewing studies of diagnostictests. Clinical Chemistry and Laboratory Medicine 38(7):577–88.

Oxman AD (1995) Checklists for review articles. In:Chalmers I, Altman DG, editors. Systematic Reviews.London: BMJ Publishing Group, p75–95.

Pai M, McCulloch M, Enanoria W et al. (2004) Systematicreviews of diagnostic test evaluations: what's behind thescenes? ACP Journal Club 141 (1):A11–3.

Pai M, McCulloch M, Gorman JD et al. (2004) Systematicreviews and meta-analysis: an illustrated, step-by-stepguide. National Medical Journal of India 17 (2):86–95.

Papadopoulos M, Rheeder P (2000) How to do asystematic literature review. South African Journal ofPhysiotherapy 56 (1):3–6.

Pawson R, Greenhalgh T, Harvey G et al. Realist synthesis:an introduction. Swindon: Economic and Social ResearchCouncil Working Paper Series. In press.

Peach H (2002) Reading systematic reviews. AustralianFamily Physician 31 (8):736–40.

Pedersen T, Moller AM, Cracknell J (2002) The mission ofthe Cochrane anesthesia review group: preparing anddisseminating systematic reviews of the effect of healthcare in anesthesiology. Anesthesia and Analgesia 95(4):1012–8.

Peersman G., Harden A, Oliver S et al. (1999)Effectiveness reviews in health promotion. London: EPPI-Centre.

Peersman G, Oliver S, Oakley A (2001) Systematic reviewsof effectiveness. In: Oliver S, Peersman G, editors. UsingResearch for Effective Health Promotion. Buckingham:Open University Press, p96–108.

Peipert JF, Bracken MB (1997) Systematic reviews ofmedical evidence: the use of meta-analysis in obstetricsand gynecology. Obstetrics and Gynecology 89(4):628–33.

Perleth M, Luhmann D, Gibis B et al. (2001) Rapidassessments – quick evaluation of medical technology.Gesundheitswesen 63 (Suppl 1):S79–84.

Petitti DB (1997) Meta-analysis and endocrinology.Endocrinology and Metabolism Clinics of North America26 (1):31–44.

Petticrew M, Song F, Wilson P et al. (1999) Quality-assessed reviews of health care interventions and thedatabase of abstracts of reviews of effectiveness (DARE).International Journal of Technology Assessment in HealthCare 15 (4):671–8.

Pogue J, Yusuf S (1998) Overcoming the limitations ofcurrent meta-analysis of randomised controlled trials.Lancet 351:47–52.

Price D, Jefferson T, Dimicheli V (2004) Methodologicalissues arising from systematic reviews of the evidence ofsafety of vaccines. Vaccine 22:2080–4.

Rees R, Elbourne D (2004) Methodological issues. In:Sebba J, Andrews R, editors. The Impact of ICT onLiteracy Education. London: Routledge Falmer, p180–201.

Rochon PA, Bero LA, Bay AM et al. (2002) Comparison ofreview articles published in peer-reviewed and throwawayjournals. Journal of the American Medical Association287 (21):2853–6.

Rosenfeld RM (1996) How to systematically review themedical literature. Otolaryngology – Head and NeckSurgery 115 (1):53–63.

Sackett DL, Straus SE, Richardson WS et al. (2000)Evidence-Based Medicine: How to Practice and TeachEBM. 2nd edition. London: Churchill Livingstone.


Sacks HS, Reitman D, Pagano D et al. (1996) Meta-analysis: an update. Mount Sinai Journal of Medicine63 (3-4):216–24.

Schlesselman JJ, Collins JA (2003) Evaluating systematicreviews and meta-analyses. Seminars in ReproductiveMedicine 21 (1):95–105.

School of Health and Related Research, University ofSheffield. ADEPT Unit 3, Reviews.www.shef.ac.uk/scharr/ir/adept/reviews/contents.htm[accessed 17 August 2005].

Shannon S (2002) Critical appraisal of systematic reviews. Canadian Association of Radiologists Journal 53 (4):195–7.

Shea B, Dube C, Moher D (2001) Assessing the quality ofreports of systematic reviews: the QUOROM statementcompared to other tools. In: Chalmers I, Egger M, SmithGD et al., editors. Systematic Reviews in Health Care:Meta-analysis in Context. 2nd edition. London: BMJ Books.

Shea B, Moher M, Graham I et al. (2002) A comparisonof the quality of Cochrane reviews and systematic reviewspublished in paper-based journals. Evaluation and theHealth Professions 25 (1):116–29.

Sheldon TA, Bradshaw J, Baldwin S et al. (2000)Methodologies for socially useful systematic reviews insocial policy. Swindon: Economic and Social ResearchCouncil.

Silagy CA, Middleton P, Hopewell S et al. (2002)Publishing protocols of systematic reviews: comparingwhat was done to what was planned. Journal of theAmerican Medical Association 287 (21):2831–4.

Sipe TA, Curlette WL (1997) A meta-synthesis of factorsrelated to educational achievement: a methodologicalapproach to summarizing and synthesizing meta-analyses.International Journal of Educational Research 25(7):583–698.

Slavin RE (1995) Best evidence synthesis: an intelligentalternative to meta-analysis. Journal of ClinicalEpidemiology 48 (1):9–18.

Smith AF (1997) An analysis of review articles publishedin four anaesthesia journals. Canadian Journal ofAnaesthesia 44 (4):405–9.

Standley JM (1996) A brief introduction to meta-analysis.Journal of Research in Music Education 44 (2):101–4.

Stevens KR (2001) Systematic reviews: the heart ofevidence-based practice. AACN Clinical Issues 12(4):529–38.

Strippoli GF, Manno C, Schena FP (2003) Critical appraisalof systematic reviews of randomised trials in nephrology:theory and practice for interpreting level I evidence.Giornale Italiano di Nefrologia 20 (6):615–24.

Stroup DF, Berlin JA, Morton SC et al. (2000) Meta-analysis of observational studies in epidemiology: aproposal for reporting. Journal of the American MedicalAssociation 283 (15):2008–12.

Stroup DF, Thacker SB, Olson CM et al. (2001)Characteristics of meta-analyses related to acceptance forpublication in a medical journal. Journal of ClinicalEpidemiology 54 (7):655–60.

Stumbo NJ (2003) Systematic reviews part I: how toconduct systematic reviews for evidence-based practice.Annual in Therapeutic Recreation 12:29–44.

Stumbo NJ (2003) Systematic reviews part II: how toappraise systematic reviews for evidence-based practice.Annual in Therapeutic Recreation 12:45–56.

Sutton AJ, Abrams KR, Jones DR et al. (1998) Systematicreviews of trials and other studies. Health TechnologyAssessment 2 (19):1–276.

Sutton AJ, Jones DR, Abrams KR et al. (1999) Systematicreviews and meta-analysis: a structured review of themethodological literature. Journal of Health ServicesResearch and Policy 4 (1):49–55.

Swann C, Bowe K, McCormick G et al. (2003) Teenagepregnancy and parenthood: a review of reviews. EvidenceBriefing. London: Health Development Agency.

Swann C, Falce C, Morgan A et al. (2003) HDA evidencebase process and quality standards manual for evidencebriefings. London: Health Development Agency.


Taylor L, Quigley R (2002) Health impact assessment: areview of reviews. Evidence Briefing. London: HealthDevelopment Agency.

Thacker SB, Peterson HB, Stroup DF (1996) Meta-analysisfor the obstetrician-gynecologist. American Journal ofObstetrics and Gynecology 174 (5):1403–7.

Thomas BH, Ciliska D, Dobbins M et al. (2004) A processfor systematically reviewing the literature: providing theresearch evidence for public health nursing interventions.World Views on Evidence-Based Nursing 1 (3):176–84.

Thomas H, Siracusa L, Ross G et al. (1999) Effectivenessof school-based interventions in reducing adolescent riskbehaviour: a systematic review of reviews. Dundas,Ontario: Effective Public Health Practice Project, Socialand Public Health Services Division, Community Supportand Research Branch Region of Hamilton-Wentworth, .

Torgerson C (2003) Systematic Reviews. London:Continuum International Publishing Group.

Tranfield D, Denyer D, Smart P (2003) Towards amethodology for developing evidence-informedmanagement knowledge by means of systematic review.British Journal of Midwifery 14:207–22.

van Tulder M, Furlan A, Bombardier C et al. (2003)Updated method guidelines for systematic reviews in theCochrane Collaboration back review group. Spine 28(12):1290–9.

Waller S, Naidoo B, Thom B (2002) Prevention andreduction of alcohol misuse. Evidence briefing. London: Health Development Agency.

West S, King V, Carey TS et al. (2002) Systems to Ratethe Strength of Scientific Evidence. EvidenceReport/Technology Assessment Number 47. Rockville:Agency for Healthcare Research and Quality.

West J, Tuffnell D, Kleijnen J (2002) Promoting evidence:systematic reviews. British Journal of Midwifery 10(12):771–5.

White A, Trinh K, Hammerschlag R (2002) Performingsystematic reviews of clinical trials of acupuncture:problems and solutions. Clinical Acupuncture and OrientalMedicine 3 (1):26–31.

White P, Lewith G, Berman B et al. (2002) Reviews ofacupuncture for chronic neck pain: pitfalls in conductingsystematic reviews. Rheumatology 41 (11):1224–31.

Whitney JD (2004) Reading and using systematic reviews.Journal of Wound, Ostomy and Continence Nursing 31(1):14–7.

Williams H, Adetugbo K, Po AL et al. (1998) TheCochrane Skin Group. Preparing, maintaining, anddisseminating systematic reviews of clinical interventionsin dermatology. Archives of Dermatology 134(12):1620–6.



APPENDIX 1

Survey of review organisations and database providers

Abstract

A survey of some UK organisations and (international)database providers was carried out in mid-2003 andupdated in December 2003 for the UK organisations. It appears that there is little consensus among databaseproviders on the terms and definitions used to describedifferent types of review article. In contrast, there is aconsensus among UK organisations that there are threemain types of review.

• Systematic review A review that adheres to anexplicit and stated method aimed at minimising biasand ensuring reliability. It may include a meta-analysis.(Some organisations specify the inclusion of the searchstrategy and explicit methods for including/excludingstudies as minimum requirements.)

• Literature review A review that does not meet thesystematic review criteria.

• Consensus group statement A summary of theevidence representing the majority agreement ofexperts using a formal method for reaching consensus.

Methods

The review terms used by the following organisations anddatabase providers were surveyed:

• Health Development Agency Evidence Base• Health Evidence Bulletins Wales• National electronic Library for Health• Health Scotland• EPPI-Centre Bibliomap• Thesaurus terms in some medical and non-medical

databases of relevance to public health.

Note on results: Most other databases using subjectheadings/thesauri use literature review as a general termand systematic review is not included (eg Care Data,PsycINFO, ERIC, British Education Index).

Organisation/database

Health Development AgencyEvidence BasePublications now at:www.publichealth.nice.org.uk

Health Evidence Bulletins Waleshttp://hebw.uwcm.ac.uk/projectmethod/index.htm

Review terms

Systematic reviews ofeffectiveness

Reviews of effectiveness

Literature reviews

Expert group reports

Systematic review

Expert opinion/review

Definitions

Analyses of ‘what works’ … Sources of both published andunpublished research are systematically searched, and the studiesfound through the search process graded accorded to the qualityof their methodology and analysis…

Similar to systematic reviews, but without the systematic searchmethodology

Interpretive reviews of the literature in specific areas

Reports from expert or working groups that will include all orsome of the above

No definition in methodology but a systematic review critical appraisalchecklist is provided – covering the CRD criteria (see Appendix 5 forcriteria). In current Bulletins, reviews that include an unbiased searchstrategy and the criteria for including studies, but which fail to meetall criteria are listed as systematic reviews with (specified) limitations.For the updated methodology (Methodology 5, in preparation) theproject team have decided that CRD criteria will have to be met in fullfor a review to be designated ‘systematic review’

All other reviews

Results [all websites accessed 23 September 2003, and updated on 10 December 2003 for UK organisations]



National electronic Library forHealthwww.nelh.nhs.uk

Instrument for preparingdescriptions of knowledgeresources[in preparation]

NHS Health Scotlandwww.phis.org.uk

EPPI-Centre Bibliomaphttp://eppi.ioe.ac.uk

[Classification provided by theEPPI-Centre. Not available on thewebsite]

MEDLINEMedical Subject Headings (MeSH)and Publication Types (pt)

EMBASEMedical Subject Headings (MeSH)

Review terms

Systematic review

Academic review

Consensus group statement

No classification in use onwebsite as yet

Systematic review

Review

[Secondary analysis]

Review literature (MeSH)

Consensus developmentconferences (MeSH – subsetof Review literature)

Review (pt)

Review academic (pt)

Review literature (pt)

Review tutorial (pt)

Review literature (MeSH) or literature (MeSH)

Definitions

…adhere to an explicit and stated method aimed at minimisingbias and ensuring reliability

...a review that does not describe an explicit and replicablemethod but may reflect the experience and views of anacknowledged authority…an ‘authoritative but not systematic’(or ‘expert’) review

...one that emerges using a formal method for reaching consensus

A systematic review is explicit in its reporting of the search forstudies (ie reports the search strategy for (a) specified database(s)and/or the list of handsearched journals) and the criteria forincluding and excluding studies; it may include meta-analysis

The authors do not attempt to include all relevant reports, nor dothey analyse any data further or in a different way to the originalreports

…re-analyses data collected in the past using new techniques toanswer the original research questions again, or using old datato answer new research questions

Review articles can cover a wide range of subject matter atvarious levels of completeness and comprehensiveness based onanalyses of literature that may include research findings

Presentations of summary statements representing the majorityagreement of physicians, scientists, and other professionalsconvening for the purpose of reaching a consensus

An article or book published after examination of publishedmaterial on a subject. It may be comprehensive to variousdegrees and the time range of material scrutinised may be broador narrow, but the reviews most often desired are reviews of thecurrent literature. The textual material examined may be equallybroad and can encompass, in medicine specifically, clinicalmaterial as well as experimental research or case reports

Work consisting of a more or less comprehensive review of theliterature on a specific subject, with usually an extensive criticalanalysis and synthesis of the literature

As MeSH definition

Work consisting of a type of review, citing literature, that gives ageneral and reasonably thorough coverage of a subject. It oftensubstitutes as a refresher course for practitioners to update theirawareness or as a quick course for a student unfamiliar with thesubject

No definition given – both terms used for medicine in literatureetc.

Results (cont.) [all websites accessed 23 September 2003, and updated on 10 December 2003 for UK organisations]



Science Citation Index Social Science Citation Index

ASSIA

Review terms

Review, Bibliography(document type)

Database review(document type)

Systematic review (occasional use in Keywordsplus)

Review articles(Thesaurus)

Literature reviews(Thesaurus)

Systematic reviews(Thesaurus)

Definitions

No definition

No definition

No definition

No definition

No definition

No definition

Results (cont.) [all websites accessed 23 September 2003, and updated on 10 December 2003 for UK organisations]

Standard search terms used for: MEDLINE, CINAHL,Cochrane Library, EMBASE, HMIC, Ovid MEDLINE In-Process & Other Non-Indexed Citations, PsycInfo.

Search terms were modified for databases where complexsearch strategies were not feasible: ASSIA, CareData,ERIC, Evidence Base, HealthPromis, NRR, SociologicalAbstracts, ISI Science Citation Index, ISI Social ScienceCitation Index, ZETOC.

Limits: Human, All Languages, All Countries, 1995–17 June 2004, All Publication Types

Set 1 – ‘Review terms’

exp "Review Literature"/exp Meta-Analysis/exp "Systematic Review"/ st, ev, mt [Standards,Evaluation, Methods]exp Technology Assessment, Biomedical/st [Standards]exp "Literature Review"/st,ev,mt [Standards, Evaluation,Methods]"process guideline:".mp.meta-ethnography.mp.health technology assessment:.mp.consensus development:.mp.systematic adj5 reviews.ti.meta?analys?s.ti.systematic overviews.ti.literature reviews.ti.systematic narrative.ti.review adj5 evidence.ti.cochrane collaboration.ti.synthesis of result:.ti.conduct: a systematic review.ti.undertak: a systematic review.ti.develop: a systematic review.ti.systematic review process.ti.scientific: evidence.ti.OR/

Set 2 ‘Quality criteria’ terms

exp DATABASES, BIBLIOGRAPHIC/st [Standards]exp Reference Databases/ev [Evaluation]exp Evidence-Based Medicine/mt [Methods]exp "BIAS (RESEARCH)"/ev [Evaluation]GUIDELINES/st [Standards]Research Design/st [Standards]"Sensitivity and Specificity"/"Bias (Epidemiology)"/quality criteri:.mp.report: of standard:.mp.report: accurac:.mp.report: of measure:.mp.definition:.mp.classification:.mp.

bibliographic: database:.mp.quality of report:.mp.methodogolical quality.ti.conduct: and reporting.ti.OR/

AND/Set 1, Set 2

Sensitivity (MEDLINE) = 16.7% (4/25 hits selected)

? = 'wildcard' that was used as a substitute for one ormore characters: = a substitute for one missing character


APPENDIX 2

Search terms

Advanced Management Research Centre (AMRC), UKAggressive Research Intelligence Facility (ARIF), UKAgency for Healthcare Research and Quality (AHRQ), USAAlberta Heritage Foundation for Medical Research,

CanadaBazian, UKBest Evidence Medical Eduction (BEME), UKBerkeley Systematic Reviews Group, USABritish Columbia Office of Health Technology Assessment,

CanadaCampbell Collaboration (Campbell Collaboration

Information Retrieval Methods Group), USACanadian Coordinating Office for Health Technology

Assessment, CanadaCentre for Evidence-Based Medicine (CEBM), UKCentre for Outcomes Research and Effectiveness

(CORE), UKCentre for Statistics in Medicine (CSM), UKChalmers Research Group (CONSORT), CanadaCentre for Reviews and Dissemination, UKCochrane Collaboration (Cochrane Information Retrieval

Methods Group), UKCochrane HIV/AIDS Group, UKCochrane Reporting Bias Methods Group, CanadaCochrane Effective Practice and Organisation of Care

Group (EPOC), CanadaCochrane Qualitative Research Methods Group and

Campbell Process Implementation Methods Group, UKCochrane Qualitative Methods Network, UKCritical Appraisal Skills Programme (CASP), UKEffective Public Health Practice Project (EPHPP), CanadaESRC UK Centre for Evidence Based Policy & Practice, UKESRC Research Methods Programme, University of

Manchester, UKEvidence-based Practice Center, Research Triangle

Institute International, University of North Carolina at Chapel Hill, USA

Evidence for Policy and Practice Information and Co-ordinating Centre (EPPI-Centre), UK

Finnish Office for Healthcare Technology Assessment, Finland

Health Services Research Unit (HSRU), UKHealth Scotland, UKInternational Centre for Evidence-Based Periodontal

Health (ICEPH), UK

International Network of Agencies for Health Technology Assessment (INAHTA), Sweden

Institute of Health Sciences, UKJoanna Briggs Institute (UK Collaborating Centres), UKMcMaster Evidence-based Practice Center, CanadaMRC Social and Public Health Sciences Unit, UKNational Coordinating Centre for Health Technology

Assessment (NCCHTA), UKNational Health and Medical Research Council (NHMRC),

AustraliaNational Institute for Health and Clinical Excellence

(NICE), UKNational Library of Medicine, USANational Training and Research Appraisal Group

(NTRAG), UKNew Zealand Health Technology Assessment:

The Clearing House for Health Outcomes and Technology Assessment (NZHTA), New Zealand

Public Health Research, Education and Development (PHRED), Canada

School of Health and Related Research (ScHARR), UKSchool of Care Sciences, University of Glamorgan, UKScottish Intercollegiate Guidelines Network (SIGN), UKSuccinct and Timely Evaluated Evidence Reviews (STEER),

New ZealandSwedish Council on Technology Assessment in Health

Care (SBU), SwedenUnit for Evidence-Based Practice and Policy, UKVA Technology Assessment Program (VATAP), USAWessex Institute for Health Research and

Development, UK


APPENDIX 3

Some UK and international organisations undertaking systematic reviews

For all multiple choice questions, please check all those responses that are appropriate

Tracking information

Reference number:

Citation:

Publication type:

Published article Abstract/presentationReportInternet website publicationUnpublished dissertation/reportBook/book chapterOther (specify):

Does the paper state that the authors have used the Oxman and Guyatt Yes Noquality criteria or methodology index?* (see references below)

1 DefinitionDoes the paper provide a definition of the term ‘systematic review’? Yes No

If yes, describe

2 RationaleDoes the paper state that a literature search should be made to ensure Yes Nothat a high-quality review in the field of interest has not been conducted, prior to undertaking the review?


APPENDIX 4

Data extraction form

* Oxman AD, Cook DJ, Guyatt GH (1994) Users’ guides to the medical literature VI. How to use an overview. Journal of the American Medical Association 272 (17):1367–71.Oxman AD, Guyatt GH (1991) Validation of an index of the quality of review articles. Journal of Clinical Epidemiology 44 (11):1271–8.Oxman AD, Guyatt GH, Singer J et al. (1991) Agreement among reviewers of review articles. Journal of Clinical Epidemiology 44 (1):91–8.Oxman AD (1994) Checklists for review articles. British Medical Journal 309:648–51.


3 ProtocolDoes the paper state that a review should include and report a pre-stated protocol? Yes No

If yes, does it state what components the protocol should specify? Steering/review group Yes NoFocused main question (and any secondary questions) Yes NoStudy inclusion/exclusion criteria (study design, types of participants) Yes NoSearch strategy Yes NoIntervention/exposure of interest Yes NoStudy selection criteria and procedures Yes NoStudy quality assessment and procedures Yes NoOutcome measures to be assessed Yes NoData analysis strategy Yes NoData extraction and synthesis of extracted data Yes NoReporting and dissemination Yes No

4 Focused questionDoes the paper state that a review should include a precisely stated question? Yes NoDoes the paper discuss/suggest the use of ‘PICO’?* Yes No

5 Study inclusion/exclusion criteriaDoes the paper state that a review should explicitly report ‘a priori’ Yes Noinclusion/exclusion criteria?

If yes, does it provide general guidance on what the inclusion/exclusion criteria should address? Population Yes NoProblem of interest Yes NoType(s) of intervention considered (and comparison intervention) Yes NoOutcomes to be considered Yes NoTypes of study designs Yes No

6 Search strategyDoes the paper state that a review should conduct and report a literature search? Yes No

If yes, does it provide general guidance on the components of the search strategy? Yes NoElectronic databases Yes No

design of the search strategy Yes Nosearch terms to be used Yes Nolist of databases to be searched Yes No

Hand searching Yes NoGrey literature Yes NoInternet Yes NoReference lists Yes NoContact with experts Yes NoYears searched Yes NoUpdate search Yes NoSpecified language constraints Yes No

* P – Patient, population or problem; I – intervention or independent variable; C – comparison; O – dependent variables or outcomes of interestSee eg Stone PW (2002) Popping the (PICO) question in research and evidence-based practice. Applied Nursing Research 15 (3):197–8.


7 Study selectionDoes the paper state that a review should report the study selection process? Yes No

If yes, does it provide general guidance on the methods and reporting of this process? Yes NoLevels of screening Yes NoApplication of inclusion/exclusion criteria Yes NoNumber of reviewers Yes NoKappa score Yes NoIncluded/excluded papers Yes No

8 Study quality assessmentDoes the paper state that a review should conduct and report a Yes Noquality assessment of studies?

If yes, does it provide general guidance on the methods and reporting of this process? Yes NoQuality appraisal instruments Yes No

development Yes Nodesign and components Yes Nopilot Yes No

Number and masking of reviewers Yes Noworking independently Yes Nobetween-reviewer disagreement Yes No

9 Data extractionDoes the paper state that a review should extract and report study results? Yes No

If yes, does it provide general guidance on the methods and reporting of this process? Yes NoData extraction form Yes No

development Yes Nodesign and content Yes Nopilot Yes No

Number and masking of reviewers Yes Noworking independently Yes Nocomparison of data Yes Nobetween-reviewer disagreement Yes No

Unpublished/missing data Yes No

10 Data synthesisDoes the paper state that a review should synthesise extracted data? Yes No

If yes, does it provide general guidance on the methods and reporting of this synthesis? Yes NoDescriptive data synthesis Yes NoQuantitative data/meta-analysis Yes No

comparisons Yes Nooutcome measures Yes Noeffect measure Yes Nostatistical model/methods used Yes No

Heterogeneity/homogeneity Yes NoSensitivity analysis Yes NoPublication bias Yes No


11 Reporting and disseminationDoes the paper provide general guidance on the structure of the final report? Yes No

Executive summary/abstract Yes NoMain text Yes No

background information Yes Noreview question(s) Yes Noreview methods Yes Nodetails of the included/excluded studies Yes Noresults of the review Yes Nodiscussion Yes Noconclusions Yes Noreferences and appendices Yes Nopeer review Yes Noreview limitations Yes Nodissemination activities Yes Noreview update Yes No

Comments



Health DevelopmentAgency Evidence BasePublications now at:www.publichealth.nice.org.uk

Health Evidence BulletinsWaleshttp://hebw.uwcm.ac.uk/projectmethod/index.htm

National electronic Libraryfor Healthwww.nelh.nhs.uk

Delphi study to develop andvalidate a tool fordescribing thecharacteristics ofknowledge resources thatare made available withinthe National electronicLibrary for Health,November 2002.Southampton: WessexInstitute for HealthResearch & Development,University of Southampton

Review terms

Systematic reviews ofeffectiveness

Reviews ofeffectiveness

Literature reviews

Expert group reports

Systematic review

Expertopinion/review

Systematic review

Academic review

Consensus groupstatement

Definitions

Analyses of ‘what works’ … Sources of bothpublished and unpublished research aresystematically searched, and the studies foundthrough the search process graded accorded to thequality of their methodology and analysis…

Similar to systematic reviews, but without thesystematic search methodology

Interpretive reviews of the literature in specific areas

Reports from expert or working groups that willinclude all or some of the above

No definition in methodology but a systematic reviewcritical appraisal checklist is provided – covering theCRD criteria.* In current bulletins, reviews that includean unbiased search strategy and the criteria forincluding studies, but which fail to meet all criteria arelisted as systematic reviews with (specified) limitations.For the updated methodology (Methodology 5, inpreparation) the project team have decided that CRDcriteria will have to be met in full for a review to bedesignated ‘systematic review’*Criteria used by the CRD for the Database of Reviews ofEffectiveness (DARE): 1 Are inclusion/exclusion criteria reported that address the reviewquestion? 2 Is there evidence of a substantial effort to search for all the relevantresearch literature 3 Is the validity of the included studies adequately assessed? 4 Is sufficient detail of the individual studies presented?www.york.ac.uk/inst/crd/faq4.htm

All other reviews

…adhere to an explicit and stated method aimed atminimising bias and ensuring reliability

...a review that does not describe an explicit andreplicable method but may reflect the experience andviews of an acknowledged authority…an‘authoritative but not systematic’ (or ‘expert’) review

...one that emerges using a formal method forreaching consensus

Results [all websites accessed 23–27 September 2004]

APPENDIX 5

Survey of review organisations and database providers – update September 2004

Update definitions

Unchanged

Unchanged

Unchanged

Unchanged

Unchanged

Unchanged

Unchanged

Unchanged

Unchanged



NHS Health Scotlandwww.phis.org.uk

EPPI-Centre Bibliomaphttp://eppi.ioe.ac.uk

Peersman G, Oliver S(1997) KeywordingStrategy. Data collection forthe BiblioMap Database.London: EPPI-Centre.

MEDLINEMedical Subject Headings(MeSH) andPublication Types (pt)

National Library ofMedicinewww.nlm.nih.gov/mesh/pubtypes2004.html

Review terms

No classification inuse on website as yet

Systematic review

Review

[Secondary analysis]

Review literature(MeSH)

Consensusdevelopmentconferences (MeSH –subset of Reviewliterature)

Review (pt)

Review academic (pt)

Review literature (pt)

Review tutorial (pt)

Definitions

A systematic review is explicit in its reporting of thesearch for studies (ie reports the search strategy for(a) specified database(s) and/or the list ofhandsearched journals) and the criteria for includingand excluding studies; it may include meta-analysis

The authors do not attempt to include all relevantreports, nor do they analyse any data further or in adifferent way to the original reports

…re-analyses data collected in the past using newtechniques to answer the original research questionsagain, or using old data to answer new researchquestions

Review articles can cover a wide range of subjectmatter at various levels of completeness andcomprehensiveness based on analyses of literaturethat may include research findings

Presentations of summary statements representingthe majority agreement of physicians, scientists, andother professionals convening for the purpose ofreaching a consensus

An article or book published after examination ofpublished material on a subject. It may becomprehensive to various degrees and the time rangeof material scrutinised may be broad or narrow, butthe reviews most often desired are reviews of thecurrent literature. The textual material examined maybe equally broad and can encompass, in medicinespecifically, clinical material as well as experimentalresearch or case reports

Work consisting of a more or less comprehensivereview of the literature on a specific subject, withusually an extensive critical analysis and synthesis ofthe literature

As MeSH definition

Work consisting of a type of review, citing literature,that gives a general and reasonably thoroughcoverage of a subject. It often substitutes as arefresher course for practitioners to update theirawareness or as a quick course for a studentunfamiliar with the subject

Results (cont.) [all websites accessed 23–27 September 2004]

Update definitions

Unchanged

Unchanged

Unchanged

Unchanged

Unchanged

Unchanged

Unchanged

Unchanged

Unchanged



EMBASEMedical Subject Headings(MeSH)http://gateway.uk.ovid.com/gw1/ovidweb.cgi

Science Citation Index andSocial Science CitationIndexhttp://wok.mimas.ac.uk

ASSIAProvided by: Aaron Share,product support specialist,Cambridge ScientificAbstracts, 7200 WisconsinAve, Suite 601, Bethesda,MD 20814

CINAHLMedical Subject Headingshttp://gateway.uk.ovid.com/gw1/ovidweb.cgi

HMIChttp://gateway.uk.ovid.com/gw1/ovidweb.cgi

Review terms

Systematic Review (MeSH)

(Date of entry 30 June 2003)

Review literature(MeSH) or literature(MeSH)

Review, Bibliography(document type)

Database review(document type)

Systematic review (occasional use inKeywords plus)

Review articles(Thesaurus)

Literature reviews(Thesaurus)

Systematic reviews(Thesaurus)

Systematic review

(MeSH)

(Year entry 1998)

Systematic reviews

(MeSH)

(Year entry notprovided)

Definitions

Scope note does not provide a definition. Used forReview, Systematic

No definition given – both terms used for medicine inliterature etc

No definition

No definition

No definition

An article, in a periodical, which reviews a number ofbooks on a particular topic in some detail

A review of the literature on a given topic, either ofthe whole of the literature or the current literature, iemore wide ranging than the Review Article

A review of a clearly formulated question that usessystematic and explicit methods to identify, selectand critically appraise relevant research, and tocollect and analyse data from the studies that areincluded in the review. Statistical methods (meta-analysis) may or may not be used to analyse andsummarise the results of the included studies. (Thisdefinition is taken from the Cochrane Database)

A specific type of literature review in which a conceptis identified and the research which has studied it isanalysed and evaluated. The results of this researchare synthesised to present the current state ofknowledge regarding the concept. For articles thatare systematic reviews or contain systematic reviews,search for document type ‘systematic review’. Whenthe MeSH term ‘Literature Review’ is exploded thisterm is captured

Reviews of literature in a particular field that has setexplicit tests for whether research is valuable enoughto be included. This is often combined with astatistical meta-analysis of clinical trial results. Theindexed term ‘Literature Reviews’ is captured whenthis term is exploded

Results [all websites accessed 23–27 September 2004]

Update definitions

Identified during theinterval between theoriginal survey(December 2003)and the currentupdate survey

Unchanged

Unchanged

Unchanged

Unchanged

Definition received,directly from CSA,during update survey






APPENDIX 6

Excluded studies

Albanese M, Norcini J (2002) Systematic reviews: What are they and whyshould we care? Advances in Health Sciences Education 7 (2):147–51.

Amato L, Davoli M, Ferri M et al. (2004) Effectiveness of interventions onopiate withdrawal treatment: an overview of systematic reviews. Drug andAlcohol Dependence 73 (3):219–26.

Ament A, Evers S, Goossens M et al. (2002) Criteria list for conductingsystematic reviews based on economic evaluation studies – the CHECproject. In: Donaldson C, Mugford M, Vale L, editors. Evidence-basedHealth Economics: From Effectiveness to Efficiency in Systematic Review.London: BMJ Books, p99-113.

Barer D (1995) Narrative or systematic reviews: can we be more‘evidence-based’? Reviews in Clinical Gerontology 4 (5):365–7.

Bland CJ, Meurer LN, Maldonado G (1995) A systematic approach toconducting a non-statistical meta-analysis of research literature. Academic Medicine 70 (7):642–53.

Casali PG, Licitra L, Bruzzi P (2000) QUOROM and the search for anupdated ‘clinical method’ in the era of evidence-based medicine. Annals of Oncology 11 (8):923–5.

Chalmers I, Hedges LV, Cooper H (2002) A brief history of researchsynthesis. Evaluation and the Health Professions 25 (1):12–37.

Clark K, Ohlsson A (2003) Cochrane systematic reviews of health careinterventions. Journal of Canadian Chiropractice Association 47 (1):8–16.

Clarke M, Ghersi D (1997) Meta-analysis, collaborative overview,systematic review: what does it all mean? Australian Prescriber20 (4):93–6.

Creugers NH, Kreulen CM (2003) Systematic review of 10 years ofsystematic reviews in prosthodontics. International Journal ofProsthodontics 16 (2):123–7.

Contributors to the Cochrane Collaboration and the CampbellCollaboration (2000) Evidence from systematic reviews of researchrelevant to implementing the ‘wider public health agenda’. York: Centre for Reviews and Dissemination.

Cummings M (2000) Teasing apart the quality and validity in systematicreviews of acupuncture. Acupuncture Medicine 18 (2):104–7.

Illustration of how a systematic review differs from atraditional review only. Systematic review processguidance was not reported.

Overview of systematic reviews. Inclusion/exclusionand quality assessment criteria not reported.

Checklist for quality validation of economicevaluations. Did not meet inclusion criteria.

General discussion of systematic review literature.Systematic review process guidance was not reported.

Systematic review. Did not meet inclusion criteria.

Commentary on QUOROM.

History and overview of research synthesis. Systematicreview process guidance was not reported.

Description of the seven major steps in the process ofa Cochrane Review taken from the CochraneReviewer’s Handbook.

Systematic review definition. Systematic review processguidance was not reported.

Inventory study and assessment of yielded evidence.Did not meet inclusion criteria.

Full methodology could not be obtained. Authors(UKCC and Centre for Reviews and Dissemination)contacted for assistance.

Discussion of the quality of trials included in onesystematic review. Did not meet inclusion criteria.

Reference Reasons for exclusion


Davey Smith G, Egger M (1998) Meta-analysis: Unresolved issues andfuture developments. British Medical Journal 316 (7126):221–5.

Davoli M, Minozzi S, Amato L et al. (2004) Monitoring the quality ofsystematic reviews published in the Cochrane Library: the experience ofthe drugs and alcohol group (CDAG). XI Cochrane Colloquium, 2004.

de Vet HC, van der Weijden T, Muris JW et al. (2001) Systematic reviewsof diagnostic research. Considerations about assessment and incorporationof methodological quality. European Journal of Epidemiology17 (4):301–6.

Dickersin K (2002) Systematic reviews in epidemiology: why are we so farbehind? International Journal of Epidemiology 31 (1):6–12.

Duley L (1996) Systematic reviews: What can they do for you? Journal of the Royal Society of Medicine 89 (5):242–4.

Edwards A, Elwyn G, Hood K et al. (2000) Judging the 'weight ofevidence' in systematic reviews: introducing rigour into the qualitativeoverview stage by assessing Signal and Noise. Journal of Evaluation inClinical Practice 6 (2):177–84.

Edwards P, Lipp A (2001) Developing a systematic review: trials andtribulations. British Journal of Perioperative Nursing 11 (11):487–90.

Ernst E (2002) A systematic review of systematic reviews of homeopathy.British Journal of Clinical Pharmacology 54: 577–82

Evans D, Pearson A (2001) Systematic reviews: gatekeepers of nursingknowledge. Journal of Clinical Nursing 10: 593–9.

Fergusson D, Laupacis A, Salmi LR et al. (2000) What should be includedin meta-analyses? An exploration of methodological issues using theISPOT meta-analyses. International Journal of Technology Assessment inHealth Care 16 (4):1109–19.

Gatchel RJ, McGeary D (2002) Cochrane collaboration-based reviews ofhealth-care interventions: are they unequivocal and valid scientifically, orsimply nihilistic? [comment]. Spine 2 (5):315–9.

Green BN, Johnson CD, Adams A (2001) Writing narrative literaturereviews for peer-reviewed journals: secrets of the trade. Journal of SportsChiropractic and Rehabilitation 15 (1):5–19.

Hailey D, Corabian P, Harstall C et al. (2000) The use and impact of rapidhealth technology assessments. International Journal of TechnologyAssessment in Health Care 16 (2):651–6.

Discussion of unpublished data in meta-analyses. Didnot meet inclusion criteria.

Insufficient data reported in abstract. Authorscontacted. Reply not obtained.

Assessment of the methodological quality of includedstudies in systematic reviews. Did not meet inclusioncriteria.

Discussion of the obstacles and difficulties ofconducting systematic reviews. Systematic reviewprocess guidance was not reported.

Discussion of the potential hazards of conducting asystematic review and meta-analysis. Systematic reviewprocess guidance was not reported.

Rationale for the use of ‘Signal’. Systematic reviewprocess guidance was not reported.

This paper reported only the initial steps in conductinga systematic review.

Overview of systematic reviews. Inclusion/exclusionand quality assessment criteria not reported.

Discussion of systematic reviews in the context ofnursing research. Systematic review process guidancewas not reported.

Study of effects of review methods on conclusions. Didnot meet inclusion criteria.

Discussion of the results and shortcomings of two‘reviews of reviews’. Systematic review processguidance was not reported.

Guidelines for preparing a narrative review for journalsubmission.

Discussion of the experiences of one HealthTechnology Agency when providing a rapidassessment. Systematic review process guidance wasnot reported.


Appendix 6 – excluded studies (cont.)


Hammersley M (2002) Systematic or unsystematic, is that the question?Some reflections on the science, art and politics of reviewing researchevidence. Text of a talk given to The UK Public Health Steering Group ofthe Health Development Agency, October 2002. London: HealthDevelopment Agency.

Hutton JL, Ashcroft R (1998) What does ‘systematic’ mean for reviews ofmethods? In: Black N, Brazier J, Fitzpatrick R et al., editors. Health servicesresearch methods: a guide to best practice. London: BMJ Books, p249–54.

LaPier TK (2003) Methods for finding systematic reviews relevant tophysical therapy: bridging research and clinical practice. Journal ofCardiopulmonary Physical Therapy 14 (1):9–12.

Lilford RJ, Richardson A, Stevens A et al. (2001) Issues in methodologicalresearch: perspectives from researchers and commissioners. HealthTechnology Assessment 5 (8):1–57.

Millward LM, Morgan A, Kelly MP (2003) Prevention and reduction ofaccidental injury in children and older people. London: HealthDevelopment Agency.

Offringa M, De Craen AJ (1999) The practice of systematic reviews. I. Introduction. Nederlands Tijdschrift voor Geneeskunde 143 (13):653–6.

Olsen O, Alderson P, Ezzo J et al. (1999) Improving the quality ofCochrane reviews – a pilot project. 7th Annual Cochrane Colloquium.Abstracts, October 1999 in Rome. Also published as: Olsen O, MiddletonP, Ezzo J et al. (2001) Quality of Cochrane reviews: assessment of samplefrom 1998. British Medical Journal 323:829–32.

Pawson R (2003) Assessing the quality of evidence in evidence-basedpolicy: why, how and when? ESRC Research Methods Programme.Working Paper No 1. Leeds: University of Leeds.

Pearson A (2004) Balancing the evidence: incorporating the synthesis ofqualitative data into systematic reviews. JBI Reports 2:45–64.

Perleth M, Jakubowski E, Busse R (2001) What is ‘best practice’ in healthcare? State of the art and perspectives in improving the effectiveness andefficiency of the European health care systems. Health Policy 56(3):235–50.

Petrie A, Bulman JS, Osborn JF (2003) Further statistics in dentistry Part 8:Systematic reviews and meta-analyses. British Dental Journal 194(2):73–8.

Poikolainen K (2002) A nice try that fails: The Swedish council ontechnology assessment in health care (SBU) evaluation of the effect oftreatment of alcohol and drugs problems: the epidemiologist's view.Alcohol and Alcoholism 37 (5):416–8.

Discussion of ‘systematic’ versus ‘unsystematic’.Systematic review process guidance was not reported.

Definition and discussion of systematic reviews.Systematic review process guidance was not reported.

This paper discussed the methods for locatingsystematic reviews. Did not meet inclusion criteria.

Qualitative study of the the views of researchersundertaking methodological research. Did not meetinclusion criteria.

Insufficient methodology reported.

General introductory discussion of systematic reviews.Systematic process guidance was not reported. Part of series.

Assessment format used by assessors did not includesystematic review quality criteria.

Assessment of research standards.

Study reviewing systems to review qualitative evidence.Did not meet inclusion criteria.

Literature review of ‘best practice’ in the Europeanhealth sector. Did not meet inclusion criteria.

General discussion of systematic reviews and meta-analyses in dentistry. Systematic review processguidance was not reported.

This paper outlined a Swedish Council on TechnologyAssessment in Health Care (SBU) systematic review oftreatments of alcohol and other addictive substances.Systematic review process guidance was not reported.




Smith LA, Oldman AD, McQuay HJ et al. (2000) Teasing apart quality andvalidity in systematic reviews: an example from acupuncture trials inchronic neck and back pain. Pain 86:119–32.

Thacker SB, Stroup DF (2002) Methods and interpretation in systematicreviews: Commentary on two parallel reviews of epidural analgesia duringlabor. American Journal of Obstetrics and Gynecology 186 (5):S78–80.

van Tulder MW, Assendelft WJJ, Koes BW et al. (1997) Method guidelinesfor systematic reviews in the Cochrane Collaboration back review groupfor spinal disorders. Spine 22 (20):2323–30.

Westwood ME, Kelly S, Berry E et al. (2002) How to undertake a clinicallyrelevant systematic review in a rapidly evolving field. Magnetic resonanceangiography. International Journal of Technology Assessment in HealthCare 18 (1):24–32.

Study of the quality and validity of included studies ina systematic review. Did not meet inclusion criteria.

This paper focuses on four components of a systematicreview. Did not meet inclusion criteria.

Updated method guidelines available.

Preliminary study to determine the full scope of asystematic review of MRA diagnostic imagingtechnology. Did not meet inclusion criteria.



Question 1

Is it desirable to seek a UK-wide consensus for the Yes Nodefinition of a systematic review?

Comments:

Question 2

Should the following quality criteria be included when reporting a systematic review?

Yes No2.1 Protocol2.2 Focused question2.3 Search strategy2.4 Years searched2.5 Study inclusion/exclusion criteria2.6 Study selection2.7 Study quality assessment2.8 Data extraction2.9 Data synthesis2.10 Are there any further criteria that you think should be included

in a report of a systematic review?

Comments:

Question 3

Is it desirable to seek a UK-wide consensus on criteria for the definition and quality standards for:

Yes No3.1 a literature review3.2 a consensus group statement3.3 a review of systematic reviews

Comments:

Please email your response to [email protected] by 10 December. Thank you for your contribution. All information will be anonymised before feeding back to other Delphi participants.


APPENDIX 7a

Systematic review terms and definitions – Delphi survey questionnaire 1

Thirty first-stage Delphi questionnaires were emailed toappropriate organisations and named parties who hadbeen involved in the earlier stages of this investigation.Three of these proved to be duplicate mailings to thesame organisation, leaving 27 valid contacts. The firstmailing yielded 12 completed questionnaires, by theagreed date, plus one reply apologising for being toobusy and one ‘out-of-office’ response explaining that therelevant person was on sabbatical leave for severalmonths. A reminder mailing produced a further fourcompleted questionnaires plus two replies apologising for

being too busy to complete the survey. The totalresponse was 20 out of 27.

Details of the replies from the 16 completed surveys areas follows. Some very interesting views were expressedand this was most helpful. Thank you very much for yoursupport. Full details of all comments received for eachquestion are given as an Appendix.


APPENDIX 7b

Systematic review terms and definitions – summary of first Delphi round

Yes No

Question 1: Is it desirable to seek a UK-wide consensus 13 3for the definition of a systematic review?Comments: 8 3

Question 2: Should the following quality criteria be included when reporting a systematic review?2.1 Protocol 13 32.2 Focused question 15 12.3 Search strategy 162.4 Years searched 162.5 Study inclusion/exclusion criteria 162.6 Study selection 162.7 Study quality assessment 15 12.8 Data extraction 14+1 query 12.9 Data synthesis 14+1 query 12.10 Are there any further criteria that you think should be 9 2


Question 3: Is it desirable to seek a UK-wide consensus on criteria for the definition and quality standards for:3.1 a literature review 5 113.2 a consensus group statement 10 63.3 a review of systematic reviews 10 7*Comments: 7 3

Table 5: Summary of first round Delphi survey results


Comments on Question 1

All comments are reported verbatim and have not beenmodified, bar minor style changes.

Three of the 16 who replied thought that it was notnecessary or useful to seek a UK consensus on thedefinition of a systematic review. Their commentsincluded:

• This exercise may help generate consensus on what isalready known but will not come up with somethingnew

• It’s a shifting concept and we don’t want to ossify it insome kind of contrived consensus

• Systematic reviews are an international andmultidisciplinary phenomenon so any definition shouldbe descriptive and record variability in practice, notprescriptive trying to pigeonhole existing variation intoa constrictive model.

This point was also endorsed by one who marked theirform ‘yes’ but actually gave a ‘yes and no’ response:

• Yes and no. Actually, instead of ‘seeking consensus’, itmight be more useful to highlight the fact that peoplehave continued to interpret the term ‘systematicreview’ any way they want, particularly in the socialscience field.

Three gave a modified ‘yes’:

• However, if this means that it will be dominated by therequirements of quantitative study, this would be lessthan helpful

• There is confusion at present over the definition of asystematic review. But any new definition needs to besufficiently flexible, for example to allow qualitativeresearch to be included

• (Yes, but): Consensus is urgently required because of thenonsensical invitations to tender currently being issued byfunding bodies! However in the present and evolvinginternational world linked by electronic communications,

an international rather than a national focus is required.The definitions already used by the Cochrane and theCampbell Collaborations are international and hence anational ‘definition’ might gain little respect. It may bethat a consensus definition for UK health and socialservices and other areas of public policy incorporates aset of possibilities, with differing levels ofcomprehensiveness or quality inclusion criteria, asspecified within the review. The most rigorous of theselevels might equate to a Cochrane or Campbell review.

The UK/European/international aspects were alsomentioned by two others:

• Yes, although with reservation; systematic reviews areoften carried out on an international basis and a UKdefinition that does not agree with internationaldefinitions would be of limited value

• This would avoid confusion and enable agreementbetween interested parties for example publishers andauthors. It would also be beneficial if this could beharmonised with European definitions so thateventually there would be worldwide consensus.

Three responses strongly supported the idea of a UK-wideconsensus:

• The whole purpose of evidence-based medicine is toreduce uncertainty. If we cannot agree on basicdefinitions then we are deluded into thinking evidence-based medicine can make a difference

• This would seem to be helpful for doers and users ofresearch – for doers as guidance to differentiatebetween the growing numbers of reviews that haveclaims to be systematic in some sense (eg scopingreviews, reviews of reviews) – for users a clear labelmay aid retrieval and selection, and act as a ‘kite mark’

• I may have missed it, but is there a wide range ofpublished UK definitions? Or is your concern thatpeople are misusing the systematic review term fortheir own ends – as they do the term ‘evidence based’!


Yes No

Question 1: Is it desirable to seek a UK wide consensus 13 3for the definition of a systematic review?Comments: 8 3


One was satisfied that our list included all necessarycriteria. Of the three who thought it unnecessary toinclude the protocol, one thought that it could perhapsgo on the web, as could the data extraction form (2.8).One thought it not essential to include a focusedquestion. All the other items were supportedunanimously although there were queries over thewording and meaning of points 2.8 and 2.9:

• Don’t understand what 2.8 or 2.9 mean. All reviews,systematic or not, include some element of ‘datasynthesis’ (usually narrative).

Some respondents commented on individual points fromthe list:

• 2.2 – Focused question: population, intervention,(comparison) outcome

• 2.10 – Sponsorship for potential conflict of interest• 2.6 – I presume you mean the results of the study

selection, eg five RCTs met inclusion criteria• 2.1 – The main elements of the protocol seem to be

covered in the items below• 2.5,6,7 – The boundaries in terms of both study question

and quality need to be clear. The terminology may vary ifone is undertaking a review of qualitative data.

Others gave more general comments:

• It rather depends what is included in each of theheadings above, particularly study quality assessment

• The question is worded quite vaguely – it is not clearwhat ‘be included’ means

• ‘Focused question’ – I think it is misleading to think ofa systematic review as only able to consider onefocused question – a systematic review may have morethan one focused question – all of them different.

Several provided detailed suggestions for additionalfeatures (2.10). These have been incorporated into thestage 2 Delphi survey.

Additional features suggested:

• As well as the study quality assessment, the reportshould include information relating to the process,who by and what methods were used to agree onjudgements of quality

• Tabulation– Heterogeneity (clinical, methodological, statistical)– Combinability– Interpretation– Risk of missing studies– Subgroup/sensitivity analysis

• See papers on realist review by Pawson et al. [RayPawson, School of Sociology and Social PolicyUniversity of Leeds] – we’ve just published a workingpaper for ESRC which you can download from itswebsite

• Funding source• Extra components of a systematic review report:

– Full evidence tables– Languages searched for– No. of titles, abstracts and articles found– How they eliminated duplicate publications from

same study


Yes NoQuestion 2: Should the following quality criteria be included when reporting a systematic review?2.1 Protocol 13 32.2 Focused question 15 12.3 Search strategy 162.4 Years searched 162.5 Study inclusion/exclusion criteria 162.6 Study selection 162.7 Study quality assessment 15 12.8 Data extraction 14+1 query 12.9 Data synthesis 14+1 query 12.10 Are there any further criteria that you think should be 9 2


– How they explored potential for publication bias– How they examined potential sources of

heterogeneity– If meta-regression done, how– Whether this systematic review used individual

publication type data– How they defined the measure of interest – odds

ratio, median effect size, headcount method etc. (with detailed comments next to the ticks as well)

• Maybe the best guide to this is the QUORUM checklist– which I think may have used a Delphi or nominalgroup method...*

• Other criteria. An absolutely basic requirement is somestatistical or other assessment of heterogeneity. Otherassessments are also good practice (of robustness; ofmoderator effects; and of publication and other smallstudy biases)

• Table of included studies.

Discussion of limitations of the systematic review

• Standardised list of potential biases.• Reviews should include a commentary on whether the

review is intended to be reproducible or not. If it is notintended to be reproducible, that’s fine, but authorsshould be explicit about it.

• I think that there should be a declaration of priorinterests which states what the review team’s priorbeliefs are about the review question. This would helpto determine whether they are likely to have usedforgiving or severe methods in exploring the reviewquestion and would help with an objective assessmentof possible biases.

• Incidentally with reference to 2.3 and 2.4 I amcurrently working on a draft article which attempts tooutline standards for the reporting of literaturesearches (STARLITE) within reviews, especiallysystematic reviews


* ’A modified Delphi technique’ was indeed used with subgroupsand at plenary sessions at the Quality of Reporting of Meta-analyses (QUOROM) Conference (Chicago 1996) to assess itemswhich might be included in a checklist of standards. (Moher D,Cook DJ et al. (1999) Improving the quality of reports of meta-analyses of randomised controlled trials: the QUORUM statement.Lancet 354:1896–900.)


Four agreed with all three statements, three disagreedwith all of them (understandably these were the threewho did not agree with the concept of seeking a rigid‘definition’). The rest gave an assortment of ‘yes’ or ‘no’answers, with one ‘yes and no’ reply to point 3.3.Comments were as follows, from those who replied ‘yes’or ‘no’ to each point.

The three ‘all no’s’ said:

• This exercise may help generate consensus on qualitystandards but will not come up with a new definition

• Again we need flexibility and pluralism of approach• I believe that there are many sources of variation when

it comes to review methods – while the resultantproduct may still be called a ‘review’ it is important tobe able to isolate the individual elements, egthoroughness of searching, thoroughness of appraisal,thoroughness of synthesis etc to build up a compositeof what the strengths or limitations of a specific revieware rather than establish standardised definitions whichmask such important differences.

A general comment from the ‘all yes’ responses was:

• I think agreed criteria for these would be useful interms of ensuring that reporting is comprehensive;however, they need to relate to other criteria in a waythat we don’t end up with a range of such standards. I am not sure how you would ensure internationalconsistency where guidelines development andsystematic review are becoming a more internationalactivity.

Comments on some of the separate items were asfollows (from those whose responses to the three itemsvaried).

From one who said ‘yes’ to item 3.1 as part of a mixedresponse:

• I call the last type of activity tertiary research!

From those who said ‘no’ to item 3.1:

• The term ‘literature review’ is a broad term that coversa wide range of approaches and levels ofcomprehensiveness of searching and quality ofincluded studies, and can happily remain so. It is muchmore urgent to gain a consensus on the term‘systematic review’ because its misuse (or varied use) inofficial documents is causing confusion and mayhem!

• Literature reviews can be found in various settings andcan be written at differing academic levels. I find itdifficult to conceive how this could be standardised. Iwould also argue that standardising it could constrainthe author’s academic thought processes at an earlystage of their work. However, it may be possible togain a consensus on a literature review within specifictypes of quantitative research.

• I think that quality aspects should be focused onsystematic reviews, as ideally all literature reviewsshould be systematic, so I think that presenting criteriafor a literature review would give them too muchcredibility.

• A literature review can be undertaken at several levels:it would be impractical to attempt to cover all.

• This exercise may help generate consensus on qualitystandards but will not come up with a new definition.

• The field of synthesis is still evolving. Producing adefinition now might stifle innovation.

From those who said ‘yes’ to item 3.2:

• (3.2 and 3.3) These are relatively new concepts (pastdecade) and influence the evidence hierarchy. It wouldbe timely to ensure that they are performed in accordwith recognised standards.

• Generally I reiterate my above comment that anyharmonisation with European and worldwidedefinitions would be productive in the current instantcommunication/electronic climate.


Question 3: Is it desirable to seek a UK-wide consensus on criteria for the definition and quality standards for:3.1 a literature review 5 113.2 a consensus group statement 10 63.3 a review of systematic reviews 10 7*Comments: 7 3


• A consensus group statement needs to have an outlineof its quality attached, and criteria giving levels of thismight give added credibility to an important element inthe development of practice guidelines etc.

From one who said ‘no’ to item 3.2 as part of a mixedresponse:

• A consensus group will set its own remit and its ownstandards of quality which may depend to some extenton the remit and also on the level of agreement withinthe group. Current expectation would be to have thatreflected in the statement, but beyond that, I see noreason to intrude on the group’s work.

From those who said ‘yes’ to item 3.3 as part of a mixedresponse (in this case, one person’s ‘yes and no’ responseto this question):

• Could be useful, but the definition would not really bethat different from that of a systematic review, as areview of systematic reviews is still a systematicreview...

• If there are no quality standards for a review ofsystematic reviews, the quality of the individualsystematic reviews considered may be completelyundermined.

From those who said ‘no’ to item 3.3:

• The definition of the standard of a systematic reviewitself is much more urgent. A review of systematicreview is essentially at the level of a ‘literature review’.A more rigorous approach will be to use theinformation from a published ‘systematic review’ toobtain the original publications and undertake a morecomprehensive systematic review. ‘Reviewingsystematic reviews’ seems to imply that the definitionof systematic review allows that the review may not becontinually updated, as it is in Cochrane and Campbellreviews. An alternative may be that the publishedreview is at a level within a spectrum of systematicreview standards where continual updating does notoccur.

Conclusion

It will be seen that responses were varied and providedmany interesting additional points for consideration. On the basis of this, the second (and final) Delphiquestionnaire has been limited to two main questions forbrief consideration. It is enclosed with this report. Thank you again for your support.

Hilary Kitcher/Lesley Sander, Information Officers,Support Unit for Research Evidence, Wales Collegeof Medicine, Biology, Life and Health Sciences,Cardiff University


The first-stage Delphi questionnaire was emailed to 27 relevant organisations or named contacts and yielded 20replies, with 16 completed questionnaires. A separate Appendix gives full details of all comments made on the variouspoints in the questionnaire, which provide very interesting reading.

1 The first question for Delphi round 2 is based on your comments. Please number in order of preference and addyour comments:


APPENDIX 8a

Systematic review terms and definitions – Delphi survey questionnaire 2

Question 1: Should a UK definition of ‘a systematic review’ be: Rating

1a Agreed for use in the UK, in UK health and social services and other areas of UK public policy1b Agreed for use in the UK and across Europe1c Agreed for use in the UK, and meeting the definitions used worldwide by the Cochrane

and Campbell Collaborations

Comments:

Yes No New ratingQuestion 2: Should the following quality criteria be included when reporting a systematic review?2.3 Search strategy 16 ------2.4 Years searched 16 ------2.5 Study inclusion/exclusion criteria 16 ------2.6 Study selection 16 ------2.2 Focused question 15 1 ------

Focused question(s)2.7 Study quality assessment 15 1 ------

Method of study quality assessment2.8 Data extraction 14+1 query 1 ------

Method of data extraction2.9 Data synthesis 14+1 query 1 ------

Method of data synthesis2.1 Protocol 13 3 ------

Published protocol2.11 Funding source or sponsorship2.12 Declaration of prior interests2.13 Standardised list of potential biases2.14 Table of included studies2.15 Statistical, or other, assessment of the heterogeneity of included studies.

Comments:

2 The main suggestions for possible additional criteria have been included in the extended criteria list below – pleaseplace your vote yes or no on the new or rephrased items listed and add any comments.

One respondent suggested that these requirements werealready met by the QUORUM statement. The QUORUMchecklist is used for correct reporting of meta-analyses ofrandomised controlled trials, which the British MedicalJournal among other journals now expects to be followedby reviewers when submitting relevant contributions. TheQUORUM checklist provides a detailed plan but can besummarised as follows. Some of the items have beenseparated here for clarity:

The QUORUM statement:

• Clinical question clearly stated• Databases and other information sources clearly listed• Years included• Publication language• Inclusion/exclusion factors• Validity assessment (selection criteria)• Data abstraction processes, number of researchers• Study characteristics• Quantitative data synthesis• Meta-analysis profile summarising trial flow• Summarise key findings and interpret.

There is also the CONSORT checklist for publication ofrandomised controlled trials (see www.consort-statement.org), favoured by more than 70 biomedicaljournals (Needleman 1999)* but this covers clinical trialsrather than systematic reviews.

Our own proposed listing incorporated as question 2 ofthe first Delphi questionnaire has only one item that wasnot included in the QUORUM checklist. This is our firstitem, 2.1 Protocol, the preliminary design proposal and itspiloting. The other main difference is that the QUORUMitem ‘Meta-analysis profile summarising trial flow’ wouldnot be relevant for systematic reviews covering otherareas than RCTs, and we are devising a checklist whichwould cover systematic reviews across all areas.

Please email your response to [email protected] by 17 December. Thank you for your contribution. Allinformation will be anonymised before feeding back toother Delphi participants.

All comments from Delphi round 1 are included as anAppendix for you to read.


* Needleman I (1999) CONSORT. Consolidated Standards ofReporting Trials. British Dental Journal 186 (5):207.

Sixteen second-stage Delphi questionnaires were emailed to those who had responded to the first Delphi round. Elevencompleted questionnaires were received by the agreed date. The total response rate was 11 out of 16.


APPENDIX 8b

Systematic review terms and definitions – summary of second Delphi round

Question 1: Should a UK definition of ‘a systematic review’ be: Rating Rating Rating1 2 3

1a Agreed for use in the UK, in UK health and social services 1 2 7and other areas of UK public policy

1b Agreed for use in the UK and across Europe 7 31c Agreed for use in the UK, and meeting the definitions used 9 1

worldwide by the Cochrane and Campbell Collaborations


By a clear majority (9 out of 11) it was agreed that aninternational approach was preferred. Comments were:

• As the whole point is to define a systematic review, if this definition is correct, then it should applyinternationally…

• I would value a clarification of the definition on aninternational basis as so many systematic reviewsreview international evidence and are the basis forinternational guidelines.

• I have listed 1c as the most coherent option. It wouldbe difficult to agree it for use across Europe without amechanism (such as Cochrane) with which to facilitateit.

• As previously mentioned reviews are an internationalphenomenon and so any definition should bedetermined more broadly. However it might be usefulto have a UK policy definition of systematic reviews notso much to constrain their definition but more to makeit explicit where other types of review cannot claim tobe systematic. So a minimum standard for definition ofa review would be more useful than a prescriptive one.

• Use in UK should cover all public policy areas.• No point in reinventing the wheel!

There was an interesting reply from one who decided notto grade the three options:

• I’ve decided on rereading this that the idea of a ‘UK’definition is not a good one – a systematic review is

just a research method. Would we want a special ‘UK’definition of, say, ‘a survey’, or ‘RCT’, or ‘focus group’?Anyway I find I can’t tick any of these – the threeoptions don’t seem to be exhaustive (there are otheroptions not listed), or exclusive.

Numbers on this chart are matched to those in question 2of the first Delphi survey. The first four criteria – searchstrategy, years searched, study inclusion/exclusion criteriaand study selection – are those which receivedunanimous approval from the first round. Those whomarked it the second time still agreed with this.

Items 2.1, 2.2, 2.7, 2.8 and 2.9 were then listed with achoice of the original wording or a revised version.

• 2.1 Protocol or published protocol• 2.2 Focused question or focused question(s) • 2.7 Study quality assessment or method(s) of study

quality assessment • 2.8 Data extraction or methods of data extraction • 2.9 Data synthesis or methods of data synthesis.

2.1 Protocol or published protocol

Responses were quite consistent here – only three peopleliked ‘published protocol’ with three actively against it,and two preferred the original version. Comments on thiswere:

• ‘Published’ protocol – what does this mean? Aprotocol is important – what does ‘published’ mean?What about an unpublished one, which has beenindependently peer-reviewed? Most protocols for trials

are not ‘published’ as such – why should reviews beany different? Surely it is more important that aprotocol is drawn up in advance, and is peer-reviewed.

• Re publishing a protocol, this is fine for a Cochranereview, but may be difficult for other groups producingsystematic reviews.

2.8 Data extraction or methods of data extraction

• 2.8 is not a ‘quality criterion’ on its own – it is notclear what this question means. Does it mean thatreporting of methods of data extraction should be oneof the quality criteria?

For the other points with alternatives, three consistentlypreferred the original version, one did not like ‘methodsof data extraction’ but preferred the new version of theother three, and four liked all the new versions, twoparticularly favouring ‘methods of data synthesis’, whichreceived a bold print ‘yes!’.

Items 2.11 to 2.15 were additional items suggested byreplies to the first questionnaire.

2.12 Declaration of prior interests

Two commented on item 2.12 – declaration of priorinterests:


Yes NoQuestion 2: Should the following quality criteria be included when reporting a systematic review?2.3 Search strategy2.4 Years searched2.5 Study inclusion/exclusion criteria2.6 Study selection2.1 Protocol 22.1a Published protocol 3 32.2 Focused question 32.2a Focused question(s) 52.7 Study quality assessment 32.7a Method(s) of study quality assessment 52.8 Data extraction 32.8a Method(s) of data extraction 4 12.9 Data synthesis 32.9a Method(s) of data synthesis 42.11 Funding source or sponsorship 82.12 Declaration of prior interests 72.13 Standardised list of potential biases 6 22.14 Table of included studies 82.15 Statistical or other assessment of the heterogeneity 8

of included studies

• Re 2.12 – whose prior interests?• Prior interests – interesting idea. Conflicts of interest

are one thing – and should be declared (as for anyjournal article). But ‘prior interests’? May be toodifficult to operationalise. It is rare that anyone starts areview as a ‘tabula rasa’ – with absolutely no priorbeliefs. The minute the review begins, from the minutethe first paper is read, a prior belief begins togerminate.

2.13 Standardised list of potential biases

Item 2.13 – Standardised list of potential biases – wasdisliked by two, commenting:

• The idea of a standardised list of potential biases couldbe used by consumers of a review to assess its validity.It should not be required from producers of reviews as(1) they are not necessarily the most objective toidentify their own biases and (2) bias is not astraightforward yes/no issue – degree of bias is asubjective call and should be the judgement of theconsumer or of an independent third party

• Don’t know what this specifically refers to.

2.14 Table of included studies

All agreed to item 2.14, Table of included studies, andthere were no comments on this.

2.15 Statistical, or other, assessment of theheterogeneity of included studies

All were in favour of this as an additional criterionalthough one commented:

• Yes, there should be a statistical test included (with theproviso that they lack strength).

One person actually wanted a further item to be addedto the criteria listing:

• Add an item, guidance on generation of inferences.

One commented on the inclusion of the QUORUMstatement for consideration as an alternative and gave avery interesting analysis:

• Good – an improvement! However the QUORUMstatement appended is sadly lacking as a checklist for

systematic reviews in general, eg ‘clinical’ isinappropriate for a policy question, ‘inclusion/exclusion’needs to be worded to include non-interventionstudies, qualitative data can be synthesised as well asquantitative. I suggest that ‘meta-analysis’ is reservedfor statistical synthesis, and the term ‘meta-synthesis’used for synthesis of qualitative results. This latter is tobe distinguished from narrative review of conclusionsby its in-depth use of detailed results across studiesusing similar methods as well as research question.


terms and definitions used by UK organisations and selected ...

Documents

Transcript of terms and definitions used by UK organisations and selected ...