R

Cba

HVMa

b

c

d

e

f

g

h

a

ARRA

KCEIDEC

U

0h

Health Policy 107 (2012) 146– 156

Contents lists available at SciVerse ScienceDirect

Health Policy

j ourna l ho me p ag e: www.elsev ier .com/ locate /hea l thpol

eview

linical guidelines in the European Union: Mapping the regulatoryasis, development, quality control, implementation and evaluationcross member states

elena Legido-Quigleya,∗, Dimitra Pantelib, Serena Brusamentoc, Cécile Knaia,anessa Salibaa, Eva Turkd,e, Meritxell Solé f, Uta Augustinb, Josip Carc,g,artin McKeea,h, Reinhard Busseb,h

London School of Hygiene and Tropical Medicine, London, UKDepartment of Health Care Management, Berlin University of Technology, Berlin, GermanyGlobal eHealth, Imperial College London, London, UKNational Institute of Public Health, Ljubljana, SloveniaDet Norske Veritas, NorwayUniversity of Barcelona, Barcelona, SpainCPG Interventional Public Health, Imperial College Healthcare NHS Trust, London, UKEuropean Observatory on Health Systems and Policies, Brussels, Belgium

r t i c l e i n f o

rticle history:eceived 4 February 2012eceived in revised form 5 June 2012ccepted 7 August 2012

eywords:linical guidelinesuropemplementationevelopmentvaluationhronic conditions

a b s t r a c t

Introduction: Clinical guidelines are advocated to improve the quality of care, especially forchronic diseases. However, the regulatory basis of clinical guidelines, their development,quality control, implementation and use as well as evaluation within countries across theEuropean Union is not systematically known.Methods: Using information collected from key informants in each country by means of astructured questionnaire, this mapping exercise illustrates the varied status of guidelineproduction in European Union countries.Results: Most European Union countries have an established national, regional or local clin-ical guideline programme, and a substantial proportion have developed guidelines on theprevention and management of chronic diseases. Several countries have mechanisms inplace to ensure the quality of scientific evidence used for the development of guidelinesis high and that the process is consistent and transparent. Others are only now taking aninterest in guideline development and are taking the first steps towards establishing waysof implementing them. The majority of countries have no legal basis for the development ofguidelines and those that have well established systems mostly implement them on a vol-untary basis. The process of guideline development varies in its degrees of decentralisation

across countries with many different types of organisations taking on this responsibility.There is general acceptance of the value of the instrument developed by the AGREE collab-oration for evaluating the methodological robustness of guidelines. However, the extent towhich guidelines are implemented in Europe is unknown, as there is no systematic datacollection and, in most countries, no structure to enable it. There are few examples of formalevaluations of the development, quality, implementation and use of guidelines.

∗ Corresponding author at: London School of Hygiene and Tropical Medicine, 15-17 Tavistock Place, London WC1H 9SH,K. Tel.: +44 02079272925; fax: +44 02079272701.

E-mail address: Helena.legido-quigley@lshtm.ac.uk (H. Legido-Quigley).

168-8510/$ – see front matter © 2012 Elsevier Ireland Ltd. All rights reserved.ttp://dx.doi.org/10.1016/j.healthpol.2012.08.004

H. Legido-Quigley et al. / Health Policy 107 (2012) 146– 156 147

Our findings call for renewed efforts to respond to the severe lack of standardized guide-line terminology and accessibility as well as rigorous studies to evaluate the relationshipbetween different ways to develop guidelines and their methodological quality, betweentheir quality and the actual implementation and usage, and finally between implementation

es.

and health outcom

1. Introduction

Clinical guidelines are one of several tools available tohealth service purchasers and providers for the improve-ment of the quality of care. The Institute of Medicine(IOM) defines clinical guidelines as “systematically devel-oped statements to assist practitioner and patient decisionsabout appropriate health care for specific clinical circum-stances” [1].

Clinical guidelines are grounded in the concept ofevidence-based medicine and constitute, together withHealth Technology Assessment, a crucial link in the trans-lation between clinical research and actual health care [2].Clinical guidelines may address various aspects of deci-sion making in clinical practice, for example the choiceof diagnostic and screening tests, the provision of medi-cal and surgical services, the duration of hospital stay andother aspects of clinical practice [3]. Their aim is to reduceinappropriate variation in clinical practice and improvethe quality of healthcare. They also support education andtraining of health professionals and help patients to makeinformed decisions.

The clinical guideline development process facilitatesthe effective translation of evidence into clinical prac-tice [4,5]. The premise is that the greater the strength ofthe evidence used, the higher the quality of the guide-lines [4], and the greater the potential for rational use ofresources and improvement of quality of healthcare [6–8].Several methodologies have been developed [5,9,10] thatallow us to assess critically the quality of clinical guidelines[11,12]. Another discussion and evaluation strand focusedon strategies to disseminate and implement clinical guide-lines. The key assumption driving that research was andstill is: The more physicians know about guidelines, themore they will use them (and the better quality of care willget).

In the late 1990s, there was a concerted effort amongacademics and health care practitioners to coordinateguideline development internationally, leading to the for-mation of the AGREE Collaboration [13,14]. Their workculminated in the publication of the validated and widelyused AGREE tool [14] which identified six domains and 23specific items for assessing guideline quality [14]. A revisedversion, AGREE II, has since been developed, with refinedterms, improved supporting documentation, and strength-ened measurement properties. Its utility has been testedamong different stakeholders and the AGREE collaborationcontinues to develop additional resources to support its use[15].

However, the way guidelines are produced, imple-mented, used and evaluated within countries – and towhich extent this is tied to regulation and organizationalset-up – is still not well understood or systematically

© 2012 Elsevier Ireland Ltd. All rights reserved.

studied. Therefore, the possibility to learn from other coun-tries or to establish best practice is quite limited. As a firststep, this paper therefore seeks to systematically explorehow clinical guidelines are being developed, checked forquality (e.g. with the AGREE instrument), implementedand evaluated across the European Union (and Norwayand Switzerland), using a conceptual framework with fivedimensions.

2. Methods

This mapping of clinical guidelines across the Euro-pean Union (and Norway and Switzerland) is based onboth published and grey literature as well as informa-tion collected from key informants in each country bymeans of a questionnaire on five key components of clin-ical guidelines: (a) regulatory basis, (b) development, (c)quality control, (d) implementation and (e) evaluation(see Box 1).

The questionnaire was developed on the basis ofliterature on the desirable characteristics of guideline pro-duction [12,14,16] and use and was piloted within theinstitutions participating in the project. It was then sentto relevant experts in all countries. Key informants wereidentified through health research and policy networks,and by direct contact with ministries of health, guide-line development organisations and professional bodies.They included experts with specialist knowledge of qual-ity improvement as well as guideline developers and users.They were identified in all countries (29) and there was atleast one respondent for each country.

Data collection was conducted by e-mail. Eighty respon-dents participated from all 29 countries. Austria (3),Belgium (7), Bulgaria (1), Cyprus (1), Czech Republic (1),Denmark (2), Estonia (2), Finland (3), France (3), Germany(2), Greece (2), Hungary (3), Ireland (1), Italy (3), Latvia(2), Lithuania (1), Luxembourg (3), Malta (6), Norway(1), the Netherlands (2), Poland (4), Portugal (3), Roma-nia (3), Slovenia (2), Slovakia (3), Spain (6), Sweden (2),Switzerland (2), the United Kingdom (England) (2), andInternational (4).

A second stage of the research involved producing aprofile for each Member State and sending this informa-tion to the country experts for their feedback. This tookplace between February and May 2012. The purpose of thisexercise was to resolve inconsistencies in the data and toincorporate recent changes in the field of clinical guidelinedevelopment since the first phase of the research. which

took place in September 2011.

Inconsistencies in the accounts received were resolved,to the extent possible, by means of triangulation with datafrom different sources.

148 H. Legido-Quigley et al. / Health Po

Box 1: Assessing clinical guidelines: five dimen-sions of the framework

a. Regulatory basis: Is there an “official” basis forguideline development and implementation in thecountry, e.g. a legal basis (possibly indirectly, e.g. aspart of national service frameworks or disease man-agement programmes), a government documentor a statement by an arms’ length body or quasi-official agency (possibly the same as for HealthTechnology Assessment)? If yes, which? If no, anyproposals to create it?

b. Development: Are clinical guidelines developedcentrally (e.g. through an arms’ length body) or ina decentralized way (e.g. by different professionalassociations or individual groups of physicians) –or does this differ for different types of guidelines,e.g. prevention vs. treatment? Are there guidelinesfor clinical guideline development (e.g. regardingthe grading of the evidence, stakeholder involve-ment, editorial independence), and if yes, by whom?Is the decentralized process coordinated (e.g. by anassociation of professional associations or an arms’length body)?

c. Quality control: Are clinical guidelines checked forquality (e.g. using the AGREE instrument) beforebeing implemented, and if yes, by whom (i.e. thesame or a different body than that developingguidelines)? Is it a requirement, and if yes, bywhom?

d. Implementation and use: Is the use of (certain)clinical guidelines mandatory (which are possiblycalled differently, e.g. “directives” or “standards”)?If yes, who regulates that? How is it enforced? If notmandatory, are there (financial) incentives to imple-ment and use guidelines, e.g. through contractsbetween purchasers and providers? Is the guide-line use promoted through IT applications or othertools?

e. Evaluation: Is the development, quality control,implementation and use of clinical guidelines (regu-larly) evaluated, and if yes, by whom on using whichcriteria? Is it a requirement, and if yes, by whom?

3

3

dhvtndwp

3

t

. Results

.1. Regulatory basis

The majority of countries have no legal basis for theevelopment of clinical guidelines and those countries thatave well established systems mostly implement them on aoluntary basis. Only a few countries have a legal mandateo develop guidelines, as is the case in Latvia (regulationo. 469 “Procedures by which clinical guidelines shall beeveloped, evaluated, registered and implemented” [17]),here this, however, does not mean that any are actuallyroduced (Table 1).

.2. Development

The process of clinical guideline development varies inhe degree of decentralization across countries, with many

licy 107 (2012) 146– 156

different types of organisations taking on this responsibil-ity. In most countries professional bodies are involved buttheir level of influence varies. Countries have been classi-fied into three broad categories. The first includes countrieswhere a national agency develops clinical guidelines, thesecond includes countries where multiple organizationscontribute to the process but a central body coordinates it.Finally, the third category includes those countries wherethere is no central coordination of the guideline develop-ment process and professional bodies often step in to fillthis void (Table 1).

3.2.1. Guidelines produced by a central agencyThree countries have one national agency entirely in

charge of a top-down endorsement of recommendations.These are the National Institute for Health and ClinicalExcellence (NICE) in England, the Current Care Editorialoffice of the Finnish Medical Association (Duodecim), andthe Scientific Council in Luxembourg.

NICE is the government funded organisation respon-sible for providing national guidance and setting qualitystandards on the promotion of good health and the preven-tion and treatment of ill health. NICE produces guidance onpublic health, health technologies (pharmaceuticals, inter-ventional procedures, devices and diagnostics) and clinicalpractice. It makes recommendations to the National HealthService (NHS), local authorities and other organisations inthe public, private and voluntary sectors on new and exist-ing medicines, treatments and procedures and on treatingand caring for people with specific diseases and condi-tions. NICE consults closely with independent committeesand individual experts working in health care, academiaand industry, as well as patients and members of the pub-lic with a background or interest in the area in question.The National Collaborating Centre (NCC) for Chronic Con-ditions, which is funded by NICE and based at the RoyalCollege of Physicians (RCP), leads on developing clinicalguidelines for the treatment of chronic conditions. TheCentre for Public Health Excellence provides guidance onservices that contribute to the prevention of chronic con-ditions and encourage good health and wellbeing.

Duodecim in Finland has produced national clinicalguidelines covering over 100 clinical entities, many of themdealing with chronic conditions (such as diabetes, asthma,rheumatoid arthritis, and a range of malignant neoplasms).All cover treatment and most also extend to prevention.

3.2.2. Centrally coordinated process with contributionfrom multiple organizations

This is a more heterogeneous group of countries withregard to clinical guideline development processes. Thecommon characteristic of countries in this category is thatwhile multiple institutions may develop guidance, thereis some level of central coordination of the process. Thiscoordination may take different forms, as illustrated in thefollowing examples: in Norway the development of offi-cial national guidelines falls under the responsibility of the

Directorate of Health, however professional associationsproduce their own guidance in parallel. A similar exam-ple is that of France, where the French National Authorityfor Health (HAS, Haute Autorité de Santé) is primarily

H. Legido-Quigley et al. / Health Policy 107 (2012) 146– 156 149

Table 1The development of clinical guidelines in European Union member states.

Country Regulatorybasis

Level ofdevelopment

Main bodies Established qualitycontrol

Implementationtools

Formalevaluation

Austria + CentralProfessionalSocieties

Institute for Quality Assurance in HealthCare

AGREE + +

Belgium + Local/regional/national

Belgian Health Care Knowledge CentreBelgian Centre for Evidence-BasedMedicine EBMPracticeNet

AGREE (variesaccording toissuing body)

+ −

Bulgaria − Central (forregulation)

Medical Societies − − −

Cyprus − Central Ministry of HealthNational Health Insurance Scheme

− − −

Czech Republic − Central National Reference Center (MoH)Center for Healthcare Quality/DASHOFER

AGREE + −

Denmark − Central/regional National Board of HealthInstitute for Rational Pharmacotherapyprovide guidance on a central level.Danish College of General Practitioners

AGREE + −

Estonia + Central Guideline Advisory BodyEstonian Health Insurance FundProfessional Societies

Unclear − −

Finland − Central DuodecimCurrent Care Centre

AGREE + −

France + Central/regional/individual

Haute Authorité de Sante Evidence gradingsystem based onstudy design

− −

Germany + CentralProfessionalAssociations

Association of the Scientific MedicalSocietiesAgency for Quality in Medicine

AGREE/DELBI + −

Greece − ProfessionalSocietiesIndividualinitiative

− − − −

Hungary − CentralProvider level

National Advisory Board of Health Care AGREE − −

Ireland − Central Health Service ExecutiveNational Clinical EffectivenessCommittee

AGREE (modified) − −

Italy + Central/regional Ministry of Health Professionalassociations Regional Agencies

AGREE + −

Latvia + Professionalsocieties

Centre of Health Economics andProfessional associations

− − −

Lithuania + Central/local Ministry of HealthNational Health Insurance FundState Pharmaceutical Control Service andMandatory Health Insurance ServiceUniversities

− − −

Luxembourg − Central Conseil ScientifiqueMinistry of Health

− − −

Malta − Provider level Mater Dei Hospital − − −Netherlands − Central

ProfessionalAssociations

National Institute for Public Health andthe EnvironmentDutch Institute for HealthcareImprovement, Dutch College of GeneralPractitioners

+ (variable) − −

Norway + Central Norwegian Directorate of HealthRegional Health AuthoritiesProfessional Associations

AGREE + −

Poland − ProfessionalAssociations

College of Family PhysiciansProfessional associations

− − −

Portugal + CentralProfessionalAssociations

General Directorate for HealthPortuguese Society of Family Medicine

− + −

Romania − Central Ministry of HealthNational Centre for Family Medicine

− − −

Slovakia − CentralProfessionalAssociations

National Institute of Quality andInnovations

− − −

Slovenia − ProfessionalAssociations

Medical Associations for variousspecialties

− − −

150 H. Legido-Quigley et al. / Health Policy 107 (2012) 146– 156

Table 1 (Continued)

Country Regulatorybasis

Level ofdevelopment

Main bodies Established qualitycontrol

Implementationtools

Formalevaluation

Spain + Central/regional Quality Agency of MoH RegionalAgencies

AGREE − −

Sweden + National/regional/local

National Board of Health and WelfareSwedish Association of Local Authoritiesand Regions (SALAR)Swedish Council on TechnologyAssessment in Health Care

unclear + +

Switzerland − ProfessionalAssociations

Professional Associations AGREE/ADAPTE(variable)

− −

United Kingdom +England Central NICE AGREE + +Scotland Central SIGN AGREE + +

E (unde

rsaGoaD

mtluMsdmnrbsa26G

isDPHCcieccSmTcug

A

Wales Central NICE

Northern Ireland Central GAIN and NICagreements)

esponsible for central guidance production, but profes-ional associations and sometimes regional authorities arelso active in the field. In a similar way, Italy’s Nationaluideline System supervises guidance production, but rec-mmendations are also produced by professional bodiesnd regional agencies. An analogous situation exists inenmark.

In Germany, clinical guidelines are produced on aultitude of conditions, including the prevention and

reatment of chronic diseases. The production of guide-ines by professional associations is coordinated by theirmbrella organization, the Association of the Scientificedical Societies (the AWMF, Arbeitsgemeinschaft der Wis-

enschaftlichen Medizinischen Fachgesellschaften). Chroniciseases in particular have been the focus of diseaseanagement programmes which have been implemented

ationwide by the statutory health insurance funds inecent years (for breast cancer since 2002; type 2 dia-etes since 2002; coronary heart disease and heart failureince 2003 and 2009 respectively; asthma since 2005;nd Chronic Obstructive Pulmonary Disease (COPD) since005). In June 2011 the database of the AWMF contained79 current guidelines. Guidelines are also collected by theerman e-Health library Arztbibliothek.

The Netherlands also has a long tradition of produc-ng clinical guidelines, which are used in primary andpecialist care, covering both prevention and treatment.ifferent organizations such as the National Institute forublic Health and the Environment, the Dutch Institute forealthcare Improvement, the Dutch Council for Quality ofare or the Dutch College of General Practitioners producelinical guidelines. Clinical guidelines are also introducedndirectly by the development and implementation of dis-ase management programmes. Additionally, in primaryare, the Dutch College of General Practitioners takes aentral role in the development of clinical guidelines.ince almost all Dutch GP’s are members, the develop-ent of clinical guidelines can be considered as centralized.

he Dutch Council for Quality of Healthcare (Regieraad)onducts research and development, implementation and

pdate of guidelines, and works on the harmonization ofuideline development in the Netherlands.

In Hungary, overall directions are given by the Nationaldvisory Board of Healthcare but providers can formulate

AGREE + +r special AGREE + +

their own specific guidance based on these directions; theimplementation of these guidelines is subsequently mon-itored by the Board. In the Czech Republic – where over250 clinical guidelines have been developed since 2006and are periodically updated, including those covering dia-betes, coronary heart disease, asthma/COPD and cancer, agovernmental and a non-governmental actor (the NationalReference Centre and the DASHOFER publishing house,respectively) collaborate to produce clinical guidelines. InAustria, most guidelines are developed by professionalassociations but there is also a national guidance initiative.

3.2.3. Multiple actors produce guidance without centralcoordination

The commonality in this group of countries is that whileindividual actors produce guidance there is no one overar-ching agency or main institution that plays a coordinatingrole. However, this is not indicative of the extent to whichguidelines are developed: In Belgium, for example, severalstructures have been developed with the scope of dis-seminating the use of clinical guidelines like for examplethe Colleges of Physicians, the Belgian Health Care Knowl-edge Centre (KCE), the Belgian Centre for Evidence-BasedMedicine (CEBAM), the EBMPracticeNet and the FederalCouncil for the Quality of Nursing. At the other end of thespectrum, practitioners in Greece must rely on their ownefforts to obtain evidence, but some medical associationshave begun to show interest in the field and to providecollections of information or translations of internationalguidelines on their websites. In Poland guidance is gener-ally produced sporadically, notably by the College of FamilyPhysicians. In Slovenia, guidelines have been produced byseveral types of institutions (professional associations, aca-demic centres, etc.) whereas in Ireland multiple levers arein place to achieve quality assurance but guidance is mostlyused in an ad hoc way and is usually taken from interna-tional literature.

3.3. Quality control

Predictably, countries with long-established processesfor guideline production have systems in place to ensuretheir quality. There is a general acceptance of the instru-ment developed by the AGREE collaboration for evaluating

ealth Po

H. Legido-Quigley et al. / H

the methodological robustness of clinical guidelines. Fivedistinctive categories were identified through the ques-tionnaires: countries where the AGREE instrument iswidely used; countries where there is no quality require-ment but the AGREE instrument is used if guideline qualityis assessed; countries that are following an adapted versionof AGREE; countries that are using other instruments, andfinally, countries where no formal processes to assess thequality of clinical guidelines exist (Table 1).

3.3.1. AGREE instrument widely usedIn England, the guideline development process is based

on the AGREE instrument and described comprehen-sively in the NICE Guidelines Manual. The stakeholderconsultation, expert reviews and the assessment by theindependent guideline review panel are all part of the val-idation process. Prior to publication, the guidelines aresubjected to an internal quality control assessment at theCentre for Clinical Practice. In Norway, clinical guidelinesare checked for quality using the AGREE instrument beforebeing implemented. The AGREE instrument is used bothduring the development process, and before implementa-tion by the Secretariat of the Directorate of Health and theNorwegian Electronic Health Library.

3.3.2. No quality requirement, but AGREE instrumentused

Denmark has no formal requirement for quality assess-ment ahead of implementation but the AGREE instrumentis often used in Health Technology Assessment. Belgiumdoes not mandate mechanisms for quality assurance eitherand there is no widely used procedure. If requested, theCEBAM will validate guidelines according to the AGREEinstrument, with a limited analysis of the content. Some-times this step is a prerequisite for funding care, forexample for clinical guidelines proposed by the KCE. InSwitzerland, the AGREE and ADAPTE instruments are usedby some guidelines producers, such as the University ofLausanne. AGREE is also used informally by some guide-lines producers in Spain.

3.3.3. Countries following an adapted version of AGREEIn Germany, guidelines co-ordinated by the German

Agency for Quality in Medicine (ÄZQ, Ärztliches Zentrumfür Qualität in der Medizin) are checked for quality beforebeing implemented. The appraisal uses the German Instru-ment for Methodological Guideline Appraisal, the DELBIchecklist (Deutsches Instrument zur Bewertung der Method-ischen Leitlinienqualität), which is based on the AGREE 1instrument, adapted to the German context. The Institutefor Quality and Efficiency in Health Care (IQWiG, Insti-tut für Qualität und Wirtschaftlichkeit im Gesundheitswesen)is developing a methodology to assess clinical practiceguidelines. In Ireland, in July 2011, the Health Informationand Quality Authority (HIQA) published draft guidance forquality criteria for clinical guidelines, which includes anadapted version of the AGREE 2 tool.

3.3.4. Countries using other instrumentsIn Austria, federal quality guidelines are subject to

an explicit validation while those developed by medical

licy 107 (2012) 146– 156 151

associations or expert groups are not. Quality of clini-cal guidelines can vary quite significantly depending onwhich medical society developed them. Both a checklist forassessing guidelines as well as online information on guide-line availability are available. In the Netherlands, methodsused differ among organizations.

3.3.5. No formal processes to assess the quality of clinicalguidelines

In Cyprus, there is no system in place to assess the qual-ity of clinical guidelines before implementation, althoughthis is under review. In Latvia, as previously noted, the legalprocess in place is not being used. In Lithuania, the devel-opment of clinical guidelines approved by the Ministry ofHealth involves consultation with medical faculties, theNational Health Insurance Fund, State Pharmaceutical Con-trol Service and Mandatory Health Insurance Service. TheSlovene MoH published a manual for clinical guidelines,which recommends the AGREE tool for quality assurance,however it is rarely used.

3.4. Implementation

The extent to which clinical guidelines are implementedin clinical practice in the different parts of Europe isunknown, as there is no systematic data collection and, inmost countries, no structure to enable it. Respondents men-tioned the existence of tools to support decision-makingfor professionals and patients, including patient edu-cation, decision-support software and virtual networks.They reported that hospitals are increasingly combiningIT-applications with clinical pathways based on clinicalguidelines, although in most cases this development is stillin its infancy, at least in the outpatient care setting (Table 1).

3.4.1. Mandatory and legal aspectsPhysicians enrolling in Disease Management Pro-

grammes (DMPs) in Germany are required to treat theirpatients according to the contracts with the health insur-ance funds which are based on the decisions of the FederalJoint Committee and include guideline-like recommenda-tions, which are regularly checked for their accordancewith the latest evidence by IQWiG. Additionally, the ÄZQproduces “national care guidelines” (NVLs) for these con-ditions to guide physicians. Also, whether or not treatmentwas according to official guidelines can be taken intoaccount in malpractice cases. In Hungary, providers areallowed to formulate clinical guidelines for their ownsetting based on the centrally issued protocols or recom-mendations but are then obliged to implement their ownformulations, a situation that precludes any meaningfulmonitoring. Similarly in Italy, even though clinical guide-lines are optional, there are some mandatory protocols,mainly in the fields of occupational health and infectiousdisease control.

3.4.2. Implementation aids

Many countries report that guidance is published on the

websites of the agencies responsible for producing and dis-seminating them and are thus made accessible to a wideaudience. In well-developed systems, implementation is

1 ealth Po

aNnlt‘sIaeog

sPelmm

pMatitoIsat

3

Ut

3

dgaii

3

aaFwgoq

wtotd

52 H. Legido-Quigley et al. / H

lso aided by IT tools and other materials. Most notably,ICE has a team of implementation consultants that workationally to encourage a supportive environment and

ocally to share knowledge and support education andraining. There are generic implementation tools such as aHow to’ guide and specific tools for every clinical guidelineuch as a costing template and a PowerPoint presentation.n Sweden, several tools to facilitate the implementationnd use of clinical guidelines exist, such as publications,ducational material, conferences, IT applications, or evenrganizational interventions. Moreover, updated clinicaluidelines are sent to each registered practitioner.

In France, clinical guidelines are available on the web-ite of HAS and of the Agency for the Safety of Healthroducts (AFSSAPS; Agence Franc aise de Sécurité Sanitairet des Produits et des Produits de Santé). In the Czech Repub-ic, several web portals assist professionals and patients in

aking appropriate decisions but few are systematicallyaintained.In Finland, guidelines are used to develop regional care

rogrammes and are integrated with an Evidence-Basededicine electronic Decision Support (EBMeDS) system,

llowing guidelines to be opened from within the elec-ronic patient record. In Belgium, an EBMeDS is beingmplemented as well. It aims to bring evidence into prac-ice by means of context-sensitive guidance at the pointf care through electronic patient records. In Portugal,T tools are combined with specialized literature andpecifically designed websites. In Germany indicator-basedpproaches are used to monitor and endorse implementa-ion.

.4.3. Financial incentivesIn some countries, such as Germany, Denmark and the

nited Kingdom, financial incentives are used to encouragehe use of clinical guidelines.

.5. Evaluation

There are few examples of formal evaluations of theevelopment, quality, implementation and use of clinicaluidelines. Some countries have made explicit efforts toddress this question, others are in the process of develop-ng evaluation plans, and others have yet to consider thessue (Table 1).

.5.1. Formal evaluation of clinical guidelinesThere are several systems for evaluation even if these

re not compulsory. For example, in Germany, even in thebsence of a national agenda, the National Academy ofamily Physicians regularly evaluates all guidelines fallingithin its scope. The implementation and utilization of

uideline-based processes are also evaluated in the contextf disease management contracts through guideline-baseduality indicator programmes.

In England, NICE produces implementation reportshich measure the uptake of specific recommendations

aken from selected pieces of guidance through the analysisf routine data. Researchers assess the uptake and effec-iveness of guidance on an ad hoc basis. In Denmark, theevelopment, quality control, implementation and use of

licy 107 (2012) 146– 156

clinical guidelines are evaluated within accreditation pro-grams for publicly funded hospitals. In Finland, the NationalInstitute for Health and Welfare (THL) is responsible forsupervising the development, quality and use of clinicalguidelines since 2011. In Sweden, the development, qual-ity control, implementation and use of clinical guidelinesare regularly evaluated by the National Board of Health andWelfare. In Italy, evaluation of clinical guideline adherenceis required by law and the National Agency for RegionalHealthcare Systems is responsible for doing so.

3.5.2. Reforms to introduce some kind of evaluation arecurrently under discussion

In Malta, there is no formal evaluation of clinicalguidelines. In the hospital context, they are periodicallyevaluated by the Clinical Guideline Coordinating Commit-tee (CGCC). In primary care no audits have been performedyet but there are plans for this to take place in thenear future. In Cyprus there are no evaluation processesin place to evaluate guideline implementation and use.Tools for evaluating implementation have been proposedwithin the new National Health Insurance Scheme (NHIS).Similarly, in Czech Republic, the evaluation protocols arebeing developed. In Hungary, there is currently a part-nership between the National Advisory Board of HealthCare (NABHC) and the National Institute for Quality- andOrganizational Development in Healthcare and Medicines(GYEMSZI) to set up evaluation processes in the sector.

3.5.3. No formal evaluations of clinical guidelinesIn the Netherlands, there is no official system of reg-

ulation of the evaluation of clinical guidelines. Sporadicresearch projects by scientists, insurers or professionalsthemselves have evaluated (methods of) quality control(see above) but there is no mechanism on implementationassessment yet. However, several studies have been pub-lished to evaluate implementation and impact of clinicalguidelines [18–21].

In Portugal it is reported that evaluation is performedby a body inside the Directorate General for Health (DGS).However, the performance indicators used in the annualaudit of family physician performance are being reviewedby the government contracting department (ACSS).

In Spain, no impact evaluation system is formally inplace; sometimes assessments are carried out by the devel-opers, but since it is not a requirement there are fewexamples. However, the Ministry of Health and someregional agencies are rolling out a series of professionaltraining programmes addressed to improve quality. More-over, the Guiasalud implementation handbook includes anevaluation chapter.

Finally, we were unable to find any evidence ofevaluation of the development, quality assurance, imple-mentation and use of clinical guidelines in Latvia,Luxembourg, Slovakia, Slovenia, Switzerland, and Ireland.

3.6. Level of engagement/cooperation in the use of

clinical guidelines

All countries included in the study are showingsome interest in developing and implementing clinical

ealth Po

H. Legido-Quigley et al. / H

guidelines. International cooperation is also taking place,illustrated by projects such as the Council of Europe’sGuideline Recommendation [22], the EU-funded AGREEguideline research project [23], and the Guidelines Inter-national Network G-I-N, a Scottish Charity coordinatingthe activities of national guideline agencies worldwide[24]. However, there is great variability between coun-tries in how they are being developed and implemented.Most have an established national, regional or local clinicalguideline programme, and a substantial proportion havedeveloped guidelines for the prevention and managementof chronic diseases. Several countries have mechanismsin place to ensure that the quality of scientific evidenceused for the development of clinical guidelines is high andthat the process is consistent and transparent. Others areincreasingly interested in guideline development and aretaking steps towards establishing systems to support theirimplementation.

In some cases, guideline development is coordinatedat the national level but implementation occurs at theregional or local level as it is the case in England. Inother cases, the process moves in the opposite directionand regional quality agencies develop clinical guidelinesthat are then adopted at a national level. The mostnotable example here is Spain, where guidelines formanagement of type 2 diabetes were developed in a col-laboration between the Spanish Ministry of Health andhealth technology assessment agencies from the differentAutonomous Communities (regions), with the latter takingthe lead.

The analysis of our country responses identifies threebroad categories of engagement in clinical guideline devel-opment. The first category includes those with “wellestablished” activities and wide experience in guidelinedevelopment and implementation. The second categoryincludes those that have introduced some guidelines andare therefore “making progress” towards having ade-quate systems in place; and the third category involvesthose countries that are either “recently adopting” someclinical guidelines or where these are “in the plan-ning stage” but where some progress has been madein the conceptualization of how this might be oper-ationalized. Despite general progress, some countriesstill seem to have a long way to go before clinicalguidelines become established in mainstream clinicalpractice.

3.6.1. Countries with “well established” activitiesThis category comprises the leaders in the development

of guidelines (i.e. England, Germany, The Netherlands,France and Belgium) and other countries that have wellestablished programmes (i.e. Norway, Finland, SwedenDenmark and Italy). All countries in this category havedeveloped specific clinical guidelines for most diseases andespecially for chronic diseases.

Sweden has clinical guidelines covering both chronic

and acute conditions (e.g. diabetes, renal failure, coronaryheart disease, cataract surgery, stroke, hip fracture and hipreplacement and malignant neoplasms). The same is trueof Norway, Denmark, and Italy.

licy 107 (2012) 146– 156 153

3.6.2. Countries “making progress” in the development ofclinical guidelines

Some countries have clinical guidelines for only for afew conditions, such as Luxembourg where the ScientificCouncil (Conseil Scientifique), comprising representatives ofthe Ministry of Health, the medical examination servicesdepartment of social insurance and associations of physi-cians and dentists, has published them on its website forcardiovascular and cerebral diseases. Latvia has guidelinesfor types 1 and 2 diabetes, COPD in primary care, treatmentof autoimmune inflammatory arthritis, early detection ofmalignant tumours by general practitioners, palliative careand management of haemophilia.

In Hungary, clinical guidelines are produced by indi-vidual hospitals within a national framework. The nationalinput comes from two newly created organizations, NAB-HCGYEMSZI, both established in 2011.

Malta has no national body charged with the task ofdeveloping and implementing clinical or public healthguidelines but interested groups of clinicians have doneso on their own initiative. Some of these guidelines coverthe management of acute exacerbations of chronic diseasesbut there are none that focus on their prevention or longterm management. There are, however, protocols for druguse, developed by the national Medicines Entitlement Unitthat restrict the use of medicines within government healthservices, and define population groups and prescriber cri-teria, some of which have implications for chronic diseasemanagement.

We were unable to identify a comprehensive suite ofclinical guidelines in other countries, although individ-ual practitioners or organizations may adopt internationalguidance. This is the case in Ireland, where this is currentlydone on an ad hoc basis. Examples include the symptomaticbreast care standards and healthcare associated infectionstandards. These guidelines have been mandated throughthe Health Information and Quality Authority, a statutorybody charges with regulating and inspecting healthcareservices in Ireland. The Health Service Executive (HSE) isnow rolling out a series of care specific programmes atnational level, each with a clinical director and each specif-ically tasked with improving and standardising care acrosshealthcare services. There are programmes for diabetes,stroke, acute medicine and elective surgery among oth-ers. Each programme will, as part of its activity, providea specific set of guidelines agreed with clinical staff. Theseprogrammes are at an advanced stage of design and a feware being rolled out now.

3.6.3. Countries “recently adopting” some guidelines orwhere these are “in the planning stage”

Several countries have recently adopted some guide-lines or have begun to develop them. For example, inSlovenia, which lacks a comprehensive suite of clinicalguidelines, practitioners may use international guidanceon an ad hoc basis. In Greece, specialist medical societieshave made recommendations for the management of dia-

betes, CHD, Asthma/COPD (primary care) and rheumatoidarthritis. Clinical guidelines developed in other countries(largely in English) are available on local medical societies’websites. However, initiatives are not coordinated and it is

1 ealth Policy 107 (2012) 146– 156

ui

tnidtgsc

4

cccttt

criphlqcaEcdaet

coAtuotpmttt

dfsdecnai

Box 2: Summary of findings

• The mapping exercise provides, to our knowledge,the most up-to-date and thorough account of thedevelopment of clinical guidelines in the EuropeanUnion, illustrating considerable diversity.

• Range of practice: We can distinguish three broadcategories of Member State engagement in clinicalguideline development. The first category includesthose with a long tradition in guideline productionand implementation, where relevant activities are anestablished aspect of service provision. The secondencompasses those that have started producing andusing clinical guidelines but for a more limited rangeof settings and conditions; finally, the third categorycombines those countries where only initial stepstowards guideline utilization can be identified withany initiatives still in the planning stage.

• Regulatory framework: The value of a legal man-date for clinical guideline use is inconclusive: certaincountries have laws that are not always imple-mented, while highly developed systems oftenfunction without any legal basis.

• Bodies responsible for guideline production: Guide-lines are usually developed by government orquasi-governmental organizations and professionalassociations, often working together. In the manycountries without a comprehensive suite of clini-cal guidelines, individuals and organizations takeguidance from elsewhere, either from pan-Europeanor American bodies or from other countries. Someinstitutions, besides having their own clinical guide-lines programmes, also engage internationally withnetworks established for the purpose of knowledgeexchange, methodological development and coordi-nation of care.

• Stakeholders: The engagement of stakeholdersis a key feature in those few countries withwell-established clinical guidelines systems and isnecessary to ensure transparency. Depending onthe context, stakeholders can include represen-tatives of professional organizations, of serviceproviders, of the pharmaceutical industry and offunding bodies; patients, their families and carersand patient representatives or organizations; aca-demics or other experts; other members of civilsociety. Their involvement varies. In some coun-tries, stakeholders are encouraged to use theirnetworks and influence to encourage implementa-tion of the clinical guideline at both national and locallevel. Overall, patient and service user organizationsappear to have little influence on the developmentof clinical guidelines.

• Quality control and evaluation: Those few orga-nizations seeking to ascertain the quality of theirguidelines often use the well-accepted AGREEInstrument, in some cases adapted to context,while others have developed their own approaches.Although some countries have made explicit effortsto appraise clinical guidelines, most countries do nothave any formal way to regularly evaluate the devel-opment, quality control, implementation and use of

54 H. Legido-Quigley et al. / H

p to individuals to gather the evidence needed to makenformed decisions.

Some countries lacking guidelines have recognizedhe need to obtain them, at least as aspirations inational plans. In Poland the National Pharmaceutical Pol-

cy (2004–2008), published in 2003 identified a need for theevelopment of ambulatory health care formularies (recep-ariusz lecznictwa ambulatoryjnego), which would containuidelines on the use of medicines in specific cases and settandards of medical treatment, taking into account theirosts. However, progress has been slow.

Box 2 provides a summary of our findings.

. Discussion

This mapping exercise illustrates the varied status oflinical guideline production in the European Union. A fewountries have well-developed systems in place to developlinical guidelines while most have only fragmentary ini-iatives led by enthusiasts, with little more than aspirationso do something more formal at some unspecified time inhe future.

Our findings on the value of a legal mandate are incon-lusive. Spencer and Walshe suggested that having a legalequirement for quality improvement strategies was anmportant driver of progress [25]. However, we find exam-les of laws that exist but are not implemented, as well asighly developed systems without any legal basis. Guide-

ines are most commonly developed by government oruasi-governmental organisations and professional asso-iations, often working together. Burgers and colleaguesnalysed 86 guidelines from 11 countries (of which 10 wereuropean) in a quest to identify predictors of high-qualitylinical practice guidelines. They found that guidelines pro-uced within a guideline programme and by governmentalgencies scored better than their counterparts. But differ-nces in guideline applicability could not be explained byhe variables studied [11].

In the many countries without a comprehensive suite oflinical guidelines, individuals and organizations may relyn international guidance, either from pan-European ormerican bodies or from other countries, especially where

here is a shared language. Thus, Slovak practitioners mayse Czech guidance. Some institutions, besides having theirwn clinical guideline programmes, also engage interna-ionally with several networks established exactly for theurpose of knowledge exchange, methodological develop-ent and coordination of care. For example, in Germany

he ÄZQ, the AWMF, the Federal Joint Committee (G-BA),he IQWIG and several more institutions are members ofhe Guidelines International Network (G-I-N).

The existence of clinical guidelines for chronic diseasesepends, to some extent, on how high a priority this isor individuals responsible for developing quality of caretrategies and/or whether there are funds available toevelop such programmes. Individual interest and lead-rship are also important drivers of the development of

linical guidelines. For example, in Slovenia there is noational body responsible for clinical guidelines, but thesere developed by groups of experts that have a professionalnterest and therefore enthusiasm for this field.

clinical guidelines.

H. Legido-Quigley et al. / Health Po

• A limitation of this exercise is that while we wereable to collect information on all countries, this infor-mation is very general and it does not allow us toascertain important practical aspects of the use ofclinical guidelines, such as barriers to implementa-tion, their impact and whether those guidelines thatare being developed are of good quality.

The engagement of stakeholders is a key feature incountries with well-established clinical guideline sys-tems. Depending on the context, stakeholders can includerepresentatives of professional organizations, of serviceproviders, of the pharmaceutical industry and of fundingbodies; patients, their families and carers and patient rep-resentatives or organizations; academics or other experts;and other members of civil society. Their involvementin guidance production varies: for example, while NICEconsults registered stakeholders throughout the guidelinedevelopment process, the HAS mostly involves them inthe review of recommendations prepared by the guidelinedevelopment group. However, overall, patient and serviceuser organizations appear to have little influence on thedevelopment of clinical guidelines.

The influence of pharmaceutical companies has beenrarely reported by respondents of the questionnaires,although literature on this topic suggests that increas-ing contact has been reported between physicians andthe pharmaceutical industry. As Choudhry et al. suggestthese interactions are relevant since clinical guidelinesare designed to influence the practice of a large num-ber of physicians [26]. Stakeholder involvement may alsostrengthen implementation. For example, NICE stakehold-ers are encouraged to use their networks and influence toencourage implementation of the clinical guideline at bothnational and local level.

There is no country where there is no form of guid-ance for practitioners. In light of the increasing attentionpaid internationally to quality improvement, this is not asurprising finding. Implementation can be facilitated byintegration with IT systems, as in the United Kingdom,Finland, Sweden and Germany [27]. NICE Pathways is anonline tool for health and social care professionals thatbrings together all related NICE guidance and associatedproducts in a set of interactive topic-based diagrams. Burg-ers and colleagues found that most guideline programmesexplicitly aimed to improve the quality and effectiveness ofhealth care, used electronic databases to collect evidence,conducted systematic reviews to analyze the evidence, andemployed consensus procedures when evidence was lack-ing [12].

Those organizations seeking to ascertain the quality oftheir guidelines typically use the AGREE-Instrument, insome cases adapted to context, while others have devel-oped their own approaches. Overall, the AGREE-Instrument

seems to be well accepted among participating countries,although there are many reports of countries where no for-mal processes to assess the quality of clinical guidelinesexist. Willingness to devote resources seems an important

licy 107 (2012) 146– 156 155

factor in whether there are mechanisms to develop andevaluate clinical guidelines. The most appropriate modelfor clinical guideline implementation will vary according tothe setting. There is also a need to produce costing reportsthat estimates the national savings and costs associatedwith implementation and a costing template that can beused to estimate the local costs and savings involved.

Much less consensus appears when looking at the eval-uation of guideline programmes. Although some countrieshave made explicit efforts to address this question, mostcountries do not have any formal way to regularly evaluatethe development, quality control, implementation and useof clinical guidelines.

5. Conclusion

This mapping exercise provides, to our knowledge, themost updated and thorough account of how clinical guide-lines operate in Europe. A limitation of this exercise isthat while we were able to collect information on all thecountries, this information is quite general and it does notallow us to ascertain important aspects of clinical guide-line development such as the barriers to implementation,their impact and whether those guidelines that are beingdeveloped are of good quality.

While a few countries have made impressive progress inthis direction, many are still relying on sporadic and opaqueprocesses. The level of sophistication, quality and trans-parency of guideline development varies substantiallyacross European countries. Mechanisms for evaluatingpublished guidance are weakest, even in countries withwell-established guidance production systems. It is clearthat research and resources need to be invested in develop-ing more appropriate and easily implementable evaluationmechanisms both for the utilisation and the appropriate-ness of guidelines in place.

Even in more developed guideline production systems,the focus is mainly on the diagnosis and treatment ofa few specific diseases. However, a more comprehensiveapproach is desirable both for ensuring best possible carebut also from a financial perspective: the combination ofrecommendations on prevention with those for diagnosisand treatment or general guidelines on factors influencingpopulation health could aim to enhance prompt interven-tions and improve health altogether.

The diverse practices regarding the development, dis-semination and implementation of clinical guidelinesreflect the different stages of development of qualityassurance mechanisms across health systems. However,knowledge exchange in this field already takes place bothinformally (with practitioners “borrowing” published guid-ance from other countries) and formally in the formof collaborating networks, such as G-I-N. There is con-siderable experience in the realm of best practice withcollaborating platforms endorsed on a European level:the European Network for Health Technology Assessment(EUnetHTA) has been quite successful in promoting the

optimization of Health Technology Assessment method-ology and transfer of knowledge [28]. A similar initiativefor clinical guideline development would greatly benefitcountries where related practices are still in their infancy

1 ealth Po

aa

oodrfqth

C

m

F

bHC2vcrtr

A

tWhw

R

[

[

[

[

[

[

[

[

[

[

[

[

[

[

[

[

[

[

[

56 H. Legido-Quigley et al. / H

nd support the establishment or amelioration of qualityssurance practices.

Finally, our mapping exercise highlights a severe lackf evaluated information on what really happens in muchf Europe. Further research is needed to support stan-ardizing of guideline terminology; development of moreigorous studies to evaluate the relationship between dif-erent ways to develop guidelines and their methodologicaluality, between their quality and the actual implementa-ion and usage, and finally between implementation andealth outcomes.

onflict of interest

We are not aware of any relationships or support whichight be perceived as conflict of interest.

unding

This paper is the result of research that was requestedy the European Commission’s Directorate–General forealth and Consumers and co-funded through the EU’s FP7ooperation Work Programme: Health (contract number42058; contract acronym EUCBCC). The European Obser-atory on Health Systems and Policies is responsible foronducting this work. The European Commission is notesponsible for the content of the paper. Responsibility forhe facts described in the report and the views expressedests entirely with the authors.

cknowledgements

We thank all questionnaire respondents who took timeo generously provide us with the requested information.

e particularly wish to thank Dr. Luisa Pettigrew for all herelp in identifying respondents within the WONCA Europeorking group.

eferences

[1] Lohr KN. Institute of Medicine activities related to the devel-opment of practical guidelines. Journal of Dental Education1990;54(11):699–704.

[2] ESF. Implementation of medical research in clinical practice,www.esf.org; 2011.

[3] Health Information and Quality Authority, I. Draft national qualityassurance criteria for clinical guidelines; 2011.

[4] McQueen MJ. Overview of evidence-based medicine: challengesfor evidence-based laboratory medicine. Clinical Chemistry2001;47(8):1536–46.

[5] Shekelle PG, Woolf SH, Eccles M, Grimshaw J. Clinical guidelines:developing guidelines. BMJ 1999;318(7183):593–6.

[6] Boluyt N, Lincke CR, Offringa M. Quality of evidence-based pediatricguidelines. Pediatrics 2005;115(5):1378–91.

[7] Woolf SH, Grol R, Hutchinson A, Eccles M, Grimshaw J. Clinical guide-lines: potential benefits, limitations, and harms of clinical guidelines.BMJ 1999;318(7182):527–30.

[8] Delgado-Noguera M, Tort S, Bonfill X, Gich I, Alonso-Coello P. Qual-

ity assessment of clinical practice guidelines for the prevention andtreatment of childhood overweight and obesity. European Journal ofPediatrics 2009;168(7):789–99.

[9] Vlayen J, Aertgeerts B, Hannes K, Sermeus W, Ramaekers D. A system-atic review of appraisal tools for clinical practice guidelines: multiple

licy 107 (2012) 146– 156

similarities and one common deficit. International Journal for Qualityin Health Care 2005;17(3):235–42.

10] Lohr KN. Guidelines for clinical practice: applications for primarycare. International Journal for Quality in Health Care 1994;6(1):17–25.

11] Burgers JS, Cluzeau FA, Hanna SE, Hunt C, Grol R. Characteris-tics of high-quality guidelines: evaluation of 86 clinical guidelinesdeveloped in ten European countries and Canada. InternationalJournal of Technology Assessment in Health Care 2003;19(1):148–57.

12] Burgers JS, Grol R, Klazinga NS, Makela M, Zaat J. Towards evidence-based clinical practice: an international survey of 18 clinicalguideline programs. International Journal for Quality in Health Care2003;15(1):31–45.

13] Burgers J, Grol R, Klazinga N, van der Bij A, Makela M, Zaat J. Inter-national comparison of 19 clinical guideline programs—a survey ofthe AGREE Collaboration. Zeitschrift fur Arztliche Fortbildung undQualitatssicherung 2003;97(1):81–8.

14] AGREE Collaboration. Development and validation of an interna-tional appraisal instrument for assessing the quality of clinicalpractice guidelines: the AGREE project. Quality and Safety in HealthCare 2003;12(1):18–23.

15] Burls A. AGREE II-improving the quality of clinical care. Lancet2010;376(9747):1128–9.

16] Alonso-Coello P, Irfan A, Sola I, Gich I, Delgado-Noguera M, RigauD, et al. The quality of clinical practice guidelines over the last twodecades: a systematic review of guideline appraisal studies. Qualityand Safety in Health Care 2010;19(6):pe58.

17] The Parliament of the Republic of Latvia. Medical Treatment Law– Valsts valodas centrs. Available from: www.vvc.gov.lv/../MedicalTreatment Law.doc; 2010 [cited 2nd of February, 2012].

18] Frijling BD, Lobo CM, Hulscher ME, Akkermans RP, BraspenningJC, Prins A, et al. Multifaceted support to improve clinical decisionmaking in diabetes care: a randomized controlled trial in generalpractice. Diabetic Medicine: A Journal of the British Diabetic Associ-ation 2002;19(10):836–42.

19] Lub R, Denig P, Van Den Berg PB, Hoogenberg K, De Jong-Van Den BergLTW. The impact of new insights and revised practice guidelines onprescribing drugs in the treatment of type 2 diabetes mellitus. BritishJournal of Clinical Pharmacology 2006;62(6):660–5.

20] van Bruggen R, Gorter KJ, Stolk RP, Verhoeven RP, Rutten GEHM.Implementation of locally adapted guidelines on type 2 diabetes.Family Practice 2008;25(6):430–7.

21] Verstappen WH, van der Weijden T, Sijbrandij J, Smeele I, Hermsen J,Grimshaw J, et al. Effect of a practice-based strategy on test orderingperformance of primary care physicians: a randomized trial. JAMA2003;289(18):2407–12.

22] Council of Europe. Recommendation of the Committee of Ministersto member states on developing a methodology for draw-ing up guidelines on best medical practices. Available from:https://wcd.coe.int/wcd/ViewDoc.jsp?id=228755&BackColor-Internet=9999CC&BackColorIntranet=FFBB55&BackColorLogged=FFAC75; 2001.

23] Burgers JS, Fervers B, Haugh M, Brouwers M, Browman G, PhilipT, et al. International assessment of the quality of clinical prac-tice guidelines in oncology using the appraisal of guidelines andresearch and evaluation instrument. Journal of Clinical Oncology2004;22(10):2000–7.

24] GIN, Guidelines International Network. Available from:http://www.g-i-n.net/; 2012 [cited 1st of February, 2012].

25] Spencer E, Walshe K. National quality improvement policies andstrategies in European healthcare systems. Quality and Safety inHealth Care 2009;18(Suppl 1):i22–7.

26] Choudhry NK, Stelfox HT, Detsky AS. Relationships between authorsof clinical practice guidelines and the pharmaceutical industry. JAMA2002;287(5):612–7.

27] Kuchler R, Steinberg B, von Hubbenet J, Welcker H, Baumann J, Stadt-muller J, et al. Comprehensive care: CVD and diabetes mellitus. Herz2007;32(8):607–17.

28] Kristensen F, Mäkelä M, Allgurin Neikter S, Rehnqvist N, Lund Håheim

L, Morland B, et al. European Network for Health Technology Assess-ment, EUnetHTA. Planning, development, and implementation of asustainable European network for health technology assessment.International Journal of Technology Assessment in Health Care2009;25(S2):107–16.