THE INTERNATIONAL JOURNAL OF HEALTH PLANNING AND MANAGEMENT

Int J Health Plann Mgmt (2014)Published online in Wiley Online Library(wileyonlinelibrary.com) DOI: 10.1002/hpm.2265

Towards medicines regulatory authorities’quality performance improvement: value forpublic health

Gordana Pejović1,2*, Jovan Filipović2, Ljiljana Tasić3 andValentinaMarinković31Medicines and Medical Devices Agency of Serbia, Belgrade, Serbia2Department for Quality Management, Faculty of Organizational Science, University ofBelgrade, Belgrade, Serbia3Department for Social Pharmacy and Pharmaceutical Legislation, Faculty of Pharmacy,University of Belgrade, Belgrade, Serbia

SUMMARY

The purpose of this article is to explore the possibility of implementing total quality management(TQM) principles in national medicines regulatory authorities in Europe to achieve all publichealth objectives. Bearing in mind that medicines regulation is a governmental function thatserves societal objectives to protect and promote public health, measuring the effective achieve-ment of quality objectives related to public health is of utmost importance. A generic TQMmodelfor meeting public health objectives was developed and was tested on 10 European nationalmedicines regulatory authorities with different regulatory performances. Participating nationalmedicines regulatory authorities recognised all TQM factors of the proposed model in imple-mented systems with different degrees of understanding. An analysis of responses was performedwithin the framework of two established criteria—the regulatory authority’s category and size.The value of the paper is twofold. First, the new generic TQM model proposes to integrate fourpublic health objectives with six TQM factors. Second, national medicines regulatory author-ities were analysed as public organisations and health authorities to develop a proper tool forassessing their regulatory performance. The paper emphasises the importance of designingan adequate approach to performance measurement of quality management systems inmedicines regulatory authorities that will support their public service missions. Copyright© 2014 John Wiley & Sons, Ltd.

KEY WORDS: medicines regulatory authority; public health; performance; generic TQM model;public organisations

INTRODUCTION

Medicine regulation is a process encompassing various activities that aim to ensurethe safety, efficacy and quality of medicines, as well as the appropriateness andaccuracy of product information. This regulation’s ultimate goal is to promote and

*Correspondence to: G. Pejović, Medicines and Medical Devices Agency of Serbia, 458 Vojvode StepeStreet, 11152 Belgrade, Serbia. E-mail: gordana.pejovic@gmail.com

Copyright © 2014 John Wiley & Sons, Ltd.

G. PEJOVIĆ ET AL.

protect public health. As medicines are special merchandise at a high technologicallevel, governmental public authorities control them in all phases of their lifecycles.Medicine’s health, economic, social and political dimensions make them a notablysensitive subject in public debate.Bearing in mind that medicines regulation is a governmental function that serves

societal objectives to protect and promote public health, the following questionarises: Are these objectives being effectively achieved?The scope, nature and practices of medicines regulation (e.g. priorities, standards

and norms, enforcement strategies, resources available and rigor of enforcement)vary from country to country. However, the overall goals are generally the same:to promote and protect public health by ensuring the safety, efficacy and quality ofmedicines and the accuracy of medicinal information. Medicines regulation is publicpolicy that restricts private sector activities to attain social goals set by a givencountry. Public health and safety concerns have forced governments to take activeroles in the pharmaceutical sector framework.A national medicines regulatory authority (NMRA) is generally founded on a

legislature that represents policy choices. A host of factors relating to authority andthe capacity for exercising such authority affects the operation of medicine regulatoryactivities. These include the following:

• the scope of the regulatory authority, including its functions and sanctions;• the structure of regulatory organisations—single or multiple agencies, line ofcommand and control, coordination and communications;

• human resources—number, qualifications, remuneration and human resourcesdevelopment;

• financing—sources, adequacy and sustainability;• the standards, procedures and guidelines that guide performance of the authorisedfunctions (Ratanawijitrasin and Wondemagegnehu, 2002).

Although the importance of NMRAs in healthcare systems is undeniable, theliterature has yet to produce studies on the level of NMRA quality performance(or quality management systems implemented in such institutions) and to recognisethe importance of continuous quality improvement in terms of regulatory activities.Conversely, NMRAs are public sector organisations, as each country’s govern-

ment usually assigns the ultimate responsibility of marketing medicines to theseinstitutions. As public organisations, NMRAs do not have increased profits as theirobjectives, but they seek to satisfy specific societal needs within the constraints ofavailable budgets. Currently, there is no reported empirical research on the impactof quality practices on quality performance among NMRAs or about the use ofinnovative quality practices that arise from a knowledge-driven perspective.There are many reported methodologies and tools that propose the use of total

quality management (TQM) as a quality management framework. Several empiricalstudies have found a relationship between TQM and performance (Easton andJarrell, 1998; Powell, 1995; Taylor and Wright, 2003). Therefore, it can be said thatTQM impacts performance. Several authors suggest that the TQM perspective viewsquality as the pivotal purpose of an organisation and TQM can be used as a topmanagement tool to improve the quality of pharmaceuticals (Mazumder et al., 2011).

Copyright © 2014 John Wiley & Sons, Ltd. Int J Health Plann Mgmt (2014)

DOI: 10.1002/hpm

TQMMODEL FOR DRUG REGULATORY AUTHORITIES

As suggested by Tari et al. (2007), TQM can influence performance to a greater extentthan the ISO 9000 standard.

The aim of this paper is to explore and present the dynamic relationship betweenTQM and NMRAs’ business operations, which are both highly scientific andtechnical in nature. In particular, the paper explores the potential implementationof TQM principles in European NMRAs to improve management skills and increaseregulatory performance levels. Following a short literature review with a briefdiscussion of the methodological framework, this paper presents a model that com-prises TQM principles and common regulatory and societal objectives across variousNMRAs. Finally, empirical applications, results, findings and suggestions for furtherresearch are discussed.

TQM PRINCIPLES AND MEDICINES REGULATORY GOVERNMENTAUTHORITIES

In the EU, a well-established regulatory system ensures that the human and veterinarymedicinal products available for patient use are acceptable in every way. Governed byEU legislation, the European medicines system comprises NMRAs from 28 EU mem-ber states and three EEA-EFTA countries (i.e. Iceland, Liechtenstein and Norway), theEuropean Commission and the EuropeanMedicines Agency. Partners in the aforemen-tioned European quality infrastructure for medicines (Pejovic et al., 2011) worktogether to ensure that human and veterinary medicines available on the EU marketare properly evaluated prior to being authorised for use. These partners also continuallymonitor medicines for safety, quality and efficacy once they come onto market. Ashealth in the EU is a primary concern, it is essential that this system continues tofunction as smoothly and efficiently as possible (Lönngren, 2005). Apart fromNMRAsin EU member states, candidate countries for EU membership are striving to fullyharmonise national pharmaceutical legislation with EU directives to achieve theregulatory performance levels set by EU member states.

Regulatory work in medicines regulatory authorities should be performed withaccountability and transparency. Public accountability requires that information onNMRA decisions is available and accessible to the public. This information mayinclude negative decisions, including cases of sanctions, recalls, denials, alterations,revocation of operating licences and public health warnings. According to the UnitedStates Pharmacopeial Convention (2005), to function properly, NMRAs need astrategic plan with a clear vision and mission—objectives, strategies and targetedtime frames for meeting objectives, expected outputs and performance indicators.A clear sense of direction, good strategy and effective implementation of strategicplans will foster success. Annual work plans and self-assessment programmes willhelp identify programme weaknesses.

Several studies assess, on a wider scale, continuous quality improvement imple-mentation in healthcare organisations, primarily in hospitals (Makai et al., 2009;Wagner et al., 2006; Wagner et al., 1999; Shaw, 2000; Lammers et al., 1996;Kazmierczak and Bogusz-Czerniewicz, 2012; Rosati, 2009) or health service institu-tions (Macinati, 2008). Several authors analysed key TQM concepts, beginning with

Copyright © 2014 John Wiley & Sons, Ltd. Int J Health Plann Mgmt (2014)

DOI: 10.1002/hpm

G. PEJOVIĆ ET AL.

top management leadership and emphasising process and customer focus. It is sug-gested that TQM implementation in service hospitals requires quality managementawareness, training and framework development, and customer awareness develop-ment (Chakravarty et al., 2001). Leadership is increasingly recognised as a potentialfactor in the success of primary care quality improvement efforts, which consider thetheory of leadership, focusing on the goals of improving patient outcomes, encour-aging teamwork and utilising available power sources (Van Hoof et al., 2012).Several papers also analyse current worldwide measures and approaches that aredesigned to ensure continuous quality improvement implementation (Counte andMeurer, 2001; Shortell et al., 1995). Some authors focus more closely on qualityevolution, which occurs in healthcare organisations (De Jonge et al., 2011), or onthe actual success of TQM projects implemented thus far in European healthcareinstitutions (Øvretveit, 2000).A common thread throughout the TQM literature is the supportive role that

information management plays in quality performance. Rodger et al. (1999) investi-gated the relationship between healthcare information management and quality perfor-mance within healthcare departments.The current generation of quality concepts is based on the approaches suggested

by Deming (1982), Crosby (1997), Donabedian (1983), Juran (1989) and Garvin(1987). In manufacturing and public healthcare, quality allows organisations topursue their own objectives. In public healthcare, where quality healthcare can beinterpreted as the capacity of healthcare elements to achieve legitimate clinical andnon-clinical goals, high quality of care allows organisations to deliver appropriate pa-tient care, achieve positive clinical outcomes, avoid unnecessary clinical complicationsand ensure that public resources are efficiently used (Kunkel and Westerling, 2006;Downey-Ennis and Harrington, 2002).If we consider NMRAs as public organisations, it is worth nothing that external

actors, usually governments, determine their requirements. However, NMRAs areresponsible for the ways in which they satisfy these requirements. This responsibilityprovides NMRAs with a certain degree of autonomy in the design of service anddelivery processes. In public organisations, the search for cost reduction is usuallymore important than quality improvements. Quality is often defined as the minimumthat governments demand, where quality improvement does not lead to an increasein customer demand (Dewhurst et al., 1999).How can the effectiveness of NMRAs be measured? As stated earlier, the main

objective of medicines regulation is ensuring the safety, efficacy, quality and rationaluse of medicines that are available to the population. Therefore, the effectiveness ofmedicines regulation should be judged according to the extent to which themedicines regulatory authority achieves this objective. One way to measure medicinesregulation effectiveness is by relating the final outcomes of medicines regulation tovarious regulatory functions and by developing measures to capture regulationperformance. The ultimate measures of the effectiveness of medicine regulationpredominantly relate to public health and can be summarised as follows.

• Available medicines are efficacious for the stated diseases and conditions, and nodubious medicines are distributed on the market (the efficacy objective).

Copyright © 2014 John Wiley & Sons, Ltd. Int J Health Plann Mgmt (2014)

DOI: 10.1002/hpm

TQMMODEL FOR DRUG REGULATORY AUTHORITIES

• Available medicines are adequately safe (the safety objective).• Available medicines possess the required quality characteristics (the qualityobjective).

• Available medicines are used in accordance with approved claims and methods(the rational-use objective).

Measures to control these activities are embodied in several regulatory functions:licensing and inspection of pharmaceutical establishments, product assessment andregistration, monitoring of medicine quality, controlling and monitoring promotionand advertising, and monitoring adverse medicine reactions (i.e. pharmacovigilance).Each of these key medicine regulatory functions is designed to accomplish the four in-terrelated objectives outlined earlier (Ratanawijitrasin and Wondemagegnehu, 2002).In evaluating the effectiveness of medicine regulatory performance, one can observethat policymechanisms are essential for producing outcomes, but they are not sufficientby themselves.

Therefore, to accomplish all objectives, one possible approach for NMRAs is toimplement continuous quality improvement (CQI) or TQM. This approach aimsfor widespread implementation of underlying CQI/TQM philosophy, approachesand tools to result in the ability to maintain and improve quality and to control costincreases. The key elements in a combined definition of CQI/TQM include continuousimprovement, customer focus, structured processes and organisation-wideparticipation (Shortell et al., 1995). Theory and practice suggest that quality manage-ment systems should be characterised by certain elements that act as catalysts toimprove quality. These elements may be viewed as the ‘core values’ that healthcareorganisations need to adopt to ensure progress in quality improvement.

Current quality models have beneficial applications in organisations. However,they do not adequately address the present strategic dynamics within organisations’TQM applications (Leonard and McAdam, 2002). The literature identifies somepotential problems and constraints for the implementation of quality managementin the public sector context. Many authors believe that these problems arise fromeither the nature of quality management or the public sector itself (Milakovich,1992; Swiss, 1992; Radin and Coffee, 1993; Ustuner and Coskun, 2004). Inaddition, Erridge et al. (1998) have described some quality approaches in the publicsector, but there is no evidence in the current literature that special examination ofNMRA quality models has been performed.

It is thus necessary to develop a generic, evaluative TQM model for this type ofpublic organisation, with a special emphasis on this model’s public health impact.

DEVELOPMENT OF A GENERIC TQM MODEL TO MEET PUBLIC HEALTHOBJECTIVES IN NATIONAL MEDICINES REGULATORY AUTHORITIES

Making the right choices related to performance quality improvement for govern-mental organisations has not been easy. In the public sector, the motivation forquality improvement is not increased profits; public sector organisations are insteadbeing faced with ever-greater demands on their diminishing funds. These organisations

Copyright © 2014 John Wiley & Sons, Ltd. Int J Health Plann Mgmt (2014)

DOI: 10.1002/hpm

G. PEJOVIĆ ET AL.

must provide the ‘best value’ for the money that they have and respond to calls forincreased accountability and transparency.Since the 1990s, TQM has emerged in public sector management, especially in

US government organisations, as a cohesive frame of reference for significantimprovement in economic and human development (Sinha, 1999). Dewhurst et al.(1999) investigated TQM implementation in public sector organisations. Theseresearchers found the main barriers to be as follows:

• Lack of definition of the customer—does the organisation serve society as a wholeor the individuals who use the services?

• Organisation rigidity—change can be difficult because of the historic nature ofpublic organisations and the current systems that ensure parity and safety.

• Few incentives to improve the satisfaction of customer needs in a monopoly situation.

A comprehensive literature review presented by Fryer et al. (2007) extracts 13critical success factors for the continuous improvement of performance quality, asidentified by different authors. This literature review highlights distinct differencesamong manufacturing, service and public sectors in understanding which factor iscritical for continuous improvement. Within the public sector, management commit-ment was universally cited as critical. Customer management, process managementand employee empowerment were key in 75% of the reviewed papers. Unlike themanufacturing and service sectors, the public sector had employee empowermentand process management at the top of this critical factor list. This difference could stemfrom the traditional bureaucratic nature of the public sector and an understanding thatthis dimension should change to reap the benefits of continuous improvement. Team-work and organisational structure were also more important in the public sector thanin the manufacturing and service sectors. Conversely, the public sector is prone toreorganisation, as ministries and governments change, frequently resulting in shifts inmanagement structures, responsibilities and roles. Along with a focus on improvementand a fast response to change, management stability was listed as one of the criticalsuccess factors for public sectors organisations.The literature review (Fryer et al., 2007) emphasised six critical success factors as

the most important for public sector organisations: management commitment, cus-tomer management, communication, process management, employee empowermentand training and learning were used as a development framework for the genericTQM model, as presented in Figure 1.

Definition of the generic TQM model based on six critical success factors

As presented, top management commitment is the first step in applying and stimulat-ing the TQM approach, and top management has to accept maximum responsibilityfor existing regulatory performance. Management must also provide the necessaryleadership to create a vision, a strategic plan and organisational values and thenmotivate all employees in this common mission. In an organisation committed to TQM,all employees should be mindful of customer management and customer satisfaction.Defining the customer can be rather complex in NMRAs, but usually the customer

is the general population (i.e. patients on a national level). Therefore, the concern of

Copyright © 2014 John Wiley & Sons, Ltd. Int J Health Plann Mgmt (2014)

DOI: 10.1002/hpm

Figure 1. Generic TQM model for NMRAs

TQMMODEL FOR DRUG REGULATORY AUTHORITIES

public organisations will not be their direct users’ individual needs but the needs ofsociety as a whole. Because patients are the end-users of medicines, all medicinesregulatory efforts should ultimately lead to patient protection. Patients groups orpublic interest groups can contribute to these efforts by participating in the develop-ment of regulatory policies and in regulatory activities. These groups can act asindependent attorney generals and protect the public from undue pressure fromindustry or politicians. However, because of the highly technical nature of medicinalproducts and information, patients need support from NMRAs and other organisa-tions to empower themselves and make appropriate contributions.

In the TQM environment proposed in the model, communication is considered inrelation to relevant organisation processes and in the context of information type andstructure. High quality information has to be readily available for effective decision-making and should be part of the organisation’s visible management system.Information availability is important to identify necessary improvements and toprovide appropriate employee incentives. Information management is essential indecision-making processes in terms of knowing the results of a regulatory decisionand taking the proper measures to correct any deviation from public healthobjectives. The second important aspect of proper information management is itsimpact on transparency level increase, which all NMRAs stakeholders constantlydemand. In the context of special NMRA information types and structures, additionalefforts should be made to assure the quality of medicinal information available to thegeneral public (i.e. patients and healthcare professionals).

In relation to process management, TQM proposes that statistical and non-statistical improvement instruments should be applied when necessary, processesneed to be free of mistakes, self-inspection should be undertaken with clear work

Copyright © 2014 John Wiley & Sons, Ltd. Int J Health Plann Mgmt (2014)

DOI: 10.1002/hpm

G. PEJOVIĆ ET AL.

instructions and processes have to be capable of and maintained with statisticalprocess control (Dewhurst et al., 1999). All of these aspects aim to achieve theregulatory requirements and public health objectives presented in Figure 1 andcost decreases.The main features of the proposed TQMmodel seek to improve work processes with

the involvement of appropriately trained regulatory experts, a focus on work processvariability to identify the causes of this variability, systematic data collection on workprocesses and the use of continuous improvement problem-solving techniques byemployee teams and cross-functional groups (Hackman and Wageman, 1995).Employee training, involvement and teamwork are of great interest to medicines

regulatory authorities. Human resources are critical for successful medicines regula-tion. Governments should employ people with specialised knowledge and skillsrequired to ensure effective medicines regulation. Employees must be individualsof integrity and should be well remunerated, as medicines regulation involvesvarious stakeholders with commercial interests who may try to exert pressure onregulation authorities to secure favourable decisions.In the proposed TQM model, the six critical success factors (mentioned earlier)

were set as a framework for achieving public health objectives. The TQM modelhas four objectives at its centre that are recognised as essential for assuring a highlevel of public health—the safety of marketed medicines, their efficacy for theproposed use, the quality that assures their stability while on the market and themedicines rational use. It is recognised that these four public health objectives werecommon to all NMRAs, as their primary mission is achieving a high level of publichealth. Therefore, this model could be considered a generic model for meeting publichealth objectives that is applicable for every NMRA.

A CASE STUDY—TESTING THE TQM MODEL ON SOME EUROPEANMEDICINES REGULATORY AUTHORITIES

On the basis of the proposed TQM model, a survey has been designed to test theproposed methodology as a management tool to realise specific quality objectives—assuring quality, safe and efficacious medicines to the general public. This surveyaimed to explore the possibility of future comprehensive research in EuropeanNMRAs, which would assess their performance in accordance with the method Saraphet al. (1989) developed or other models that quality award organisations currently use.Therefore, the tool applied was a specially designed questionnaire, consisting of

three sections (Appendix A). The survey was designed to provide information abouthow public organisations (e.g. NMRAs) are organised, with no attempt to controlconditions or manipulate variables. This choice stems from the descriptive natureof this survey, which is also used to explore aspects of research questions, to seekexplanations and to provide data for testing hypotheses (Kelley et al., 2003). Thefirst section aimed to collect general information about the participating NMRAs.The second section was designed to investigate the reasons for quality managementsystem implementation, along with the benefits and challenges met during imple-mentation. This section contains a number of questions that require Likert-scaled

Copyright © 2014 John Wiley & Sons, Ltd. Int J Health Plann Mgmt (2014)

DOI: 10.1002/hpm

TQMMODEL FOR DRUG REGULATORY AUTHORITIES

responses in addition to several multiple choice questions that allowed participantsto mark more than one answer in the given set of answers. Therefore, five of sixcritical factors defined in the TQM model were described in this section—leadership,strategic planning, human resources management, information and data manage-ment, and employee participation and support for continuous improvement. Themodel’s critical factors were translated into declarative sentences, which participantscould score using the Likert-scale system. The aim was to investigate the implemen-tation of these TQM principles in regular work in the participating NMRAs.Customer management, the sixth factor described in the model, was excluded fromthe survey because customer management and customer orientation were integratedinto public health objectives. As described earlier, public sector organisations considersociety as a whole as their customer. In the case of NMRAs, the four public healthobjectives describe customer orientation, and all of these objectives should be met.

The third section of the questionnaire examined the understanding of speciallytailored benchmarking methodology (Benchmark of European Medicines Agencies(BEMA)), which was established by the Heads of Medicines Agencies.1 Thismethodology aims to develop a world-class medicines regulatory system that isbased on a network of agencies operating in accordance with the best practicestandards. BEMA assesses systems and processes in individual regulatory authoritiesagainst a set of indicators that have been agreed upon in the following areas:management systems, assessment of marketing authorisation applications,pharmacovigilance (i.e. medicines safety) activities and inspection services. BEMAis not an audit designed to identify non-compliance but an opportunity to identifyagencies’ strengths and best practices and any opportunities for improvement. Theexercise is not designed to directly compare or rank regulatory authorities. Regula-tory authorities are encouraged to install suitable best practices to enable improvedoperations in the network of regulatory authorities. Therefore, this survey aimed toanalyse the capability of NMRAs to accept BEMA methodology as their internalmanagement tool for continuous regulatory and public service performanceimprovement, as the process management dimension of TQM requires constantself-inspection to achieve effective, mistake-proof processes.

A group of academic experts examined the questionnaire for content validity andclarity. These researchers suggested a number of minor changes to clarify technicalvocabulary in questions with a Likert-scale scoring system, and these changes wereincorporated into the survey instrument.

Sample

The European NMRA network consists of over 40 competent authorities in 28member states, which includes different organisational models—agencies, bureaus orsectors within national ministries of health. Apart from the NMRAs of EU memberstates, this survey included regulatory authorities in countries that are candidates orpotential candidates for EU membership, mainly in Southeast Europe.

1Heads of Medicines Agencies. ‘Benchmarking of European Medicines Agencies (BEMA)’, available athttp://www.hma.eu/300.html (accessed 24 July 2012).

Copyright © 2014 John Wiley & Sons, Ltd. Int J Health Plann Mgmt (2014)

DOI: 10.1002/hpm

G. PEJOVIĆ ET AL.

Because the purpose of this survey was to test the defined TQM model and toexplore the potential development of a special instrument for future comprehensiveevaluation of TQM practices in NMRAs, the total number of potential participantswas chosen to be 10, with participating NMRAs of different performance levels.Therefore, to achieve a better analysis of responses, participating NMRAs weredivided in four categories:

• Category A: regulatory authorities with the highest involvement in EU regulatoryprocedures (based on European Commission Report, 2010). Three NMRAs fromthis category participated in the survey.

• Category B: regulatory authorities with high involvement in EU regulatory proce-dures. Two NMRAs from this category participated in the survey.

• Category C: regulatory authorities of member states that joined the EU in 2004.Three NMRAs from this category participated in the survey.

• Category D: regulatory authorities of candidates or potential candidates for EUaccession. Two NMRAs from this category participated in the survey.

Because purposive sampling was chosen for this study, a comprehensive study ofavailable NMRA contacts has been conducted to recognise NMRA generalmanagers and those responsible for quality management. After initial contacts weremade, the research instrument was e-mailed to the general managers and qualitymanagers of identified European regulatory authorities, as the survey requests thatparticipants have specific knowledge and experience in the field of quality manage-ment. The survey cover letter contained a note assuring respondents about theconfidentiality of the information that they were asked to provide. Therefore, thispaper contains only general information about participating NMRAs, without anyreference to the names of participating organisations. In addition, the letter addressedthe nature of the study.

RESULTS AND DISCUSSION

The survey was initiated in October 2010, and after being re-sent to non-respondents, it was completed in February 2011, with all 10 completed surveyscollected. The majority of responding NMRAs employ more than 200 employees,as presented in Table 1. All participating organisations had implemented qualitysystems to meet ISO 9001:2008 requirements, while the quality management systemwas certified in seven NMRAs. This lack of certification generally occurred in NMRAsin Categories C and D, which had lower levels of regulatory performance. However,most of the responding NMRAs that implemented ISO 9001:2008 had also imple-mented additional quality management systems: quality management systems for theirtesting laboratories (ISO 17025:2005), for their inspection activities (ISO 17020:1998)and for information security management systems (ISO 27001:2009).The agency category and size were the two criteria applied in the response analyses.Table 2 presents the analysis of reasons for and benefits from quality management

system (QMS) implementation, in relation to agency size and category. The results

Copyright © 2014 John Wiley & Sons, Ltd. Int J Health Plann Mgmt (2014)

DOI: 10.1002/hpm

Table 1. Profile and categories of participating NMRAs

Size No.NMRAcategory*

ISO 9001 certified

Yes No

Small (≤50 employees) 1 D 1 0Medium (50–200) 3 C and D 0 3Large (≥200) 6 A, B and C 6 0

*Categories of NMRA:Category A: regulatory authorities with the highest involvement in EU regulatory procedures(based on European Commission Report, 2010). In this category, three NMRAs responded tothe survey.Category B: regulatory authorities with high involvement in EU regulatory procedures. In thiscategory, two NMRAs responded to the survey.Category C: regulatory authorities of those member states that joined EU in 2004. In thiscategory, three NMRAs responded to the survey.Category D: regulatory authorities of candidates or potential candidate countries for EU acces-sion. In this category, two NMRAs responded to the survey.

TQMMODEL FOR DRUG REGULATORY AUTHORITIES

are presented as the total number of responses per question, as participants had thepossibility to mark more than one answer.

Most of the participating European regulatory authorities (82.6%) stated that themain reason for QMS implementation was to improve performance quality. Otherimportant reasons included ‘ISO 9001 is a good management tool for continuousimprovement’ (60.9%) and ‘to become respectable regulatory agency’ (39.1%). Afew NMRAs stated that they were implementing QMS to satisfy stakeholderrequests (13%) and to reduce operational costs (4.3%). Again, the respondents couldmark more than one answer to these questions.

Using the multiple-answer option, participants identified better managementcontrol (87%) as the primary benefit of QMS implementation. In addition,improvement of scientific assessment performance (78.3%), efficiency improve-ment (73.9%) and better understanding of problems related to internal proce-dures (69.6%) were identified as significant implementation benefits. Otherbenefits included the following: usefulness in new employee education(56.5%), better understanding of problems related to internal communication(43.5%), improvement of communication with stakeholders (30.4%) andoperational cost reduction (8.7%). Apart from the provided set of answers, par-ticipants could indicate other benefits of QMS implementation. The followingimportant benefits were provided:

• QMS is a foundation for IT system projects (NMRA in Category B);• harmonisation with EU inspection procedures is achieved (NMRA in Category A);• QMS enables learning from mistakes and continuous improvement (NMRA inCategory B);

• every employee should participate in the system and use its enablers in the practice(NMRA in Category C);

• improvement in performance consistency and the application of regulatoryrequirements. Application of output and outcome measures (NMRA in Category A).

Copyright © 2014 John Wiley & Sons, Ltd. Int J Health Plann Mgmt (2014)

DOI: 10.1002/hpm

Table 2. Analysis of reasons and benefits from QMS implementation in relation to size andcategory of NMRA

What are the reasonsfor QMS

implementation?What are the benefits from QMSimplementation in your institution?

A B C D E F a b c d e f g h i

Size oforganisation(number ofemployees)

Up to 20 1 0 1 1 0 0 1 1 1 0 1 1 0 1 1100–150 3 1 2 2 0 1 3 2 2 0 2 3 0 2 0Morethan 200

6 0 4 6 1 3 6 3 6 3 6 5 2 4 3

Category ofNMRA*

A 6 0 3 5 1 3 6 3 4 2 4 3 1 2 2B 2 1 2 1 0 0 2 1 2 1 1 2 0 1 2C 3 2 1 3 0 1 3 3 2 2 3 3 0 1 0D 2 0 2 2 0 0 2 2 1 0 1 1 0 2 0

*Categories of NMRA:A: regulatory authorities with the highest involvement in EU regulatory procedures (based onEuropean Commission Report, 2010). In this category, three NMRAs responded to the survey.B: regulatory authorities with high involvement in EU regulatory procedures. In this category,two NMRAs responded to the survey.C: regulatory authorities of those member states that joined EU in 2004. In this category, threeNMRAs responded to the survey.D: regulatory authorities of candidates or potential candidate countries for EU accession. Inthis category, two NMRAs responded to the survey.Reasons for QMS implementation:(a) improvement of performance quality(b) this was a demand from our stakeholders(c) to become respectable regulatory agency(d) ISO 9001 is a good management tool for continuous improvement(e) operational costs reduction(f) otherBenefits from QMS implementation in the respondent NCA:(a) better management control(b) better understanding of problems related to internal communication(c) better understanding of problems related to internal procedures(d) improvement of communication with stakeholders(e) efficiency improvement(f) assessment performance improvement(g) operational costs reduced(h) it is useful in new employees education(i) other

G. PEJOVIĆ ET AL.

On the basis of these results, it can be concluded that most regulatory authoritieshave chosen to implement QMS principles to improve the quality of regulatory per-formance, along with other aspects related to the efficiency and quality of regulatoryfunctions. On the other hand, QMS is still not recognised as a management tool foroperational cost reduction, as only 8.7% of the respondents marked this option onthe questionnaire, and this option was only marked by NMRAs with more than200 employees in Category A. This finding implies that NMRAs with higher regu-latory performance levels recognise QMS principles on a broader scale and use theseprinciples for cost reduction.

Copyright © 2014 John Wiley & Sons, Ltd. Int J Health Plann Mgmt (2014)

DOI: 10.1002/hpm

TQMMODEL FOR DRUG REGULATORY AUTHORITIES

National medicines regulatory authorities also indicated the most commonchallenges met during QMS implementation: 56.5% of all respondents foundit was an additional burden alongside regular work, and 43.5% of participantsbelieved the time needed for implementation was quite long. In addition, asignificant number of participants (26.1%) indicated that implementation costswere very high.

Respondents were asked to provide personal opinions and beliefs aboutimplementing quality management system principles using 5-point Likert scale(1 (absolutely not true); 3 (true); 5 (absolutely true)). Figure 2 presents the meanscores of the participants’ opinions about QMS principles implementation.

This survey recognises the presence of all investigated TQM critical factors in theparticipating NMRAs. However, responsible managers had different opinions aboutthe implemented systems, making this topic fertile ground for further analysis usingthe proposed TQM model.

The third section of the survey only targeted the NMRAs of EU member states, asEU membership is the precondition for BEMA procedure implementation.Therefore, only eight respondents commented in this section (the two remainingparticipants were NMRAs of EU candidate countries). The participants evaluatedseven positive aspects of BEMA methodology (Appendix A) by scoring them from1, the most beneficial aspect of BEMA methodology, to 7, the least beneficial aspect.Figure 3 presents the evaluation results. Self-assessment of the NMRAs’ regulatoryperformance, which is performed prior to BEMA visit, is evaluated as the mostbeneficial aspect with a mean score of 3.06. Participants also recognised thepossibility for continual improvement (3.47), the opportunity to evaluate thelevel of the implemented QMS (3.65) and the opportunity to obtain comprehen-sive picture of agency’s overall regulatory performance (3.76) as importantaspects. On the other hand, the respondents evaluated teamwork as the leastbeneficial aspect (5.06). Other aspects with low scores were internal comprehen-sive preparation and work prior to BEMA visit (4.53) and cooperation withcolleagues from other EU agencies (4.47).

Figure 2. Participant’s attitude towards QMS principles implementation in respondent NMRAs,presented as mean score

Copyright © 2014 John Wiley & Sons, Ltd. Int J Health Plann Mgmt (2014)

DOI: 10.1002/hpm

Figure 3. Evaluation of BEMA methodology positive aspects

G. PEJOVIĆ ET AL.

The most important negative aspects of BEMA methodology are as follows: insome sections, the BEMA questionnaire is unclear (47.8%), and the evaluationsystem was not described well (21.7%). Other negative aspects were not evaluatedto be significant. These two negative aspects indicate the necessity to furtherimprove the quality of the BEMA methodology, to achieve higher compliance inthe subjected NMRAs.However, every participating NMRA supports further organisation of BEMA

visits, as NMRAs recognise BEMA methodology as very useful towards continuousimprovement. This methodology principally seeks to create a regulatory environ-ment in the EU, where performance levels of national NMRAs will be morebalanced and all evaluation reports can be accepted in other NMRAs. This goal isa response to the European Commission’s continuous struggle to provide uniformhealthcare protection across the EU and to assure that all NMRAs meet the fourpublic health objectives mentioned in the proposed TQM model.On the basis of the results of the present survey, the European medicines

regulatory authorities that participated in testing the model recognise TQM factorsin their implemented systems. A significant number of participants recognisedquality management system as a tool to improve efficiency and scientific assessmentperformance. This recognition is very important from the public health perspective,as these aspects are directly related to the four public health objectives—quality,safety, efficacy and rational use.The survey results support the proposed TQM model for meeting the public health

objectives, as the results confirmed that the participating NMRAs implemented andpositively scored all investigated TQM factors. Consequently, a call for furtherstudies in this domain can be made, and future analysis should be performed to testthe proposed TQM model on every European NMRA, and an advanced measuringinstrument should be developed for this purpose.

CONCLUSIONS

TQM is a strategy for continuous improvement and change. However, the extent towhich various public organisations adapt TQM values, principles and practices isextremely variable.

Copyright © 2014 John Wiley & Sons, Ltd. Int J Health Plann Mgmt (2014)

DOI: 10.1002/hpm

TQMMODEL FOR DRUG REGULATORY AUTHORITIES

National medicines regulatory authorities in Europe are under increasingpressure to be accountable for and transparent about the quality of the regulatoryactivities they deliver. This study contributes to the literature and practice inregulatory performance, as no information has been previously published on thistopic. In addition, this paper has discussed the importance of NMRAs in termsof public health protection.

This study has used the proposed TQM model for meeting the public health objec-tives in European NMRAs as a framework to assess how these public organisationsunderstand TQM principles and to test whether these principles are considereduseful for the continuous improvement of regulatory functions. Six critical TQMprinciples were selected for the model, as a framework to ensure that all NMRAswould meet four necessary public health objectives. As the public health objectivesoutlined earlier are general in nature, the proposed TQM model can be considered asa generic model that applies to every NMRA.

The survey results showed that there is a high level of understanding and imple-mentation of the investigated TQM factors in the participating NMRAs, even thoughit was administered to a limited number (10) of European NMRAs. Further evalua-tion of the quality management system, based on an advanced measuring instrumentfor all European NMRAs, is the next step in our research for the immediate future.

An additional conclusion from the study results confirms that NMRAs recognisethe BEMA methodology as useful for continuous improvement of regulatoryperformance. In addition, some shortcomings of the existing BEMA methodologywere identified, and these failings should be observed as valuable input to helpresponsible EU authorities to amend and improve the existing questionnaire and toreconsider additional trainings for all interested parties applying BEMA methodo-logy. The BEMA scoring system has also proven problematic, and efforts shouldbe made to create a better and a more transparent system in the near future.

The study results indicate that NMRA managers exhibit definite signs of eager-ness to move quickly towards constant improvement and enactment of regulatoryprocesses; these managers simultaneously strive to meet all public health objectives.

ACKNOWLEDGEMENT

The authors acknowledge the contribution of the European regulatory authorities whoparticipated in this survey.

The authors have no competing interests.

REFERENCES

Chakravarty A, Parmar NK, Ranyal RK. 2001. Total quality

management — the new paradigm in health care man-

agement. Med J Armed Forces India 57(3): 226–229.

Counte M, Meurer S. 2001. Issues in the assessment of con-

tinuous quality improvement implementation in health

care organizations. Int J Qual Health Care 13: 197–207.

Copyright © 2014 John Wiley & Sons, Ltd.

Crosby PB. 1997. Quality is Free: The Art of Making

Quality Certain. MacGraw-Hill: New York.

De Jonge V, Sint Nicolaas J, Van Leerdam M, Kuipers E.

2011. Overview of the quality assurance movement in

health care. Best Pract Res Clin Gastroenterol 25(3):

337–347.

Int J Health Plann Mgmt (2014)

DOI: 10.1002/hpm

G. PEJOVIĆ ET AL.

DemingWE. 1982.Out of the Crisis. Cambridge University

Press: Cambridge, MA.

Dewhurst F, Martinez-Lorente A, Dale B. 1999. TQM in

public organizations: an examination of the issues.

Manag Serv Qual 9(4): 265–274.

Donabedian A. 1983. The Criteria and Standard of

Quality. Health Administration Press: Ann Harbor.

Downey-Ennis K, Harrington D. 2002. Organizational

effectiveness in Irish health-care organizations. Manag

Serv Qual 12(5): 316–322.

Easton GS, Jarrell SL. 1998. The effects of total quality

management on corporate performance: an empirical

investigation. J Bus 71: 253–307.

Erridge A, Fee R, McIlroy J. 1998. Public sector quality:

political project or legitimate goal?. Int J Public Sector

Manage 11(5): 341–353.

European Commission. 2010. Evaluation of the European

Medicines Agency. EC: Brussels.

Fryer K, Antony J, Douglas A. 2007. Critical success fac-

tors of continuous improvement in the public sector: a

literature review and some key findings. TQM Mag

19(5): 497–517.

Garvin DA. 1987. Competing on the eight dimension of

quality. Harv Bus Rev 65(6): 101–109.

Hackman RJ, Wageman R. 1995. Total quality manage-

ment: empirical, conceptual and practical issues.

Admin Sci Q 40(2): 309–342.

Juran J. 1989. Juran on Planning for Quality. WI:

Milwaukee.

Kaźmierczak D, Bogusz-Czerniewicz M. 2012. Identifica-

tion of patient’s requirements in quality management

system in health care institutions. Rep Practical Oncol

Radiother 17(1): 50–53.

Kelley K, Clark B, Brown V. 2003. Good practice in the

conduct and reporting of survey research. Int J Qual

Health Care 15: 261–266.

Kunkel ST, Westerling R. 2006. Different types and

aspect of quality systems and their implications. A

thematic comparison of seven quality systems at a

university hospital. Health Policy 76: 125–133.

Lammers JC, Cretin S, Giman S, Calingo E. 1996. Total

quality management in hospitals: the contribution of

commitment, quality council, teams, budgets, and

training to perceived improvement at Veterans Health

Administration hospitals. Med Care 34: 463–478.

Leonard D, McAdam R. 2002. The strategic placement of

TQM in the organization: a grounded study. Manag

Serv Qual 12(1): 43–53.

Lönngren T. 2005. The European Medicines Agency:

preparing the ground for the future. Pharm Policy

Law 6: 69–72.

Macinati M. 2008. The relationship between quality

management systems and organizational performance

in the Italian National Health Service. Health Policy

85: 228–241.

Copyright © 2014 John Wiley & Sons, Ltd.

Makai P, Klazinga N, Wagner C. 2009. Quality manage-

ment and patient safety: survey results from 102

Hungarian hospitals. Health Policy 90: 175–180.

Mazumder B, Bhattacharya S, Yadav A. 2011. Total

quality management in pharmaceuticals: a review. Int

J Pharm Tech Res 3(1): 365–375.

Milakovich ME. 1992. Total quality management for

public service productivity improvement. In Public

Productivity Handbook, Halzer M (ed). Marcel Dakke:

New York; 577–603.

Øvretveit J. 2000. Total quality management in European

healthcare. Int J Health Care Qual Assur 13(2): 74–80.

Pejović G, Filipović J, Tasić L. 2011. How to remove

barriers to medicines trade in emerging economies:

the role of medicines regulatory authority in Serbia.

Accred Qual Assur 16: 253–261.

Powell TC. 1995. Total quality management as competi-

tive advantage: a review and empirical study. Strat

Manag J 16: 15–37.

Radin B, Coffee J. 1993. A critique of TQM: problems of

implementation in the public sector. Publ Admin Q

17(1): 42–53.

Ratanawijitrasin S, Wondemagegnehu E. 2002. Effective

Drug Regulation: A Multicountry Study and Annex 1:

Guide for Data Collection to Assess Drug Regulatory

Performance. WHO: Geneva.

Rodger J, Pendharkar P, Paper D. 1999. Management of

information technology and quality performance in

health care facilities. Int J Appl Qual Manag 2(2):

251–269.

Rosati R. 2009. The history of quality measurement in

home health care. Clin Geriatr Med 25(1): 121–134.

Saraph JV, Benson GP, Schroeder RG. 1989. An

instrument for measuring the critical factor of quality

management. Decis Sci 20: 810–829.

Shaw CD. 2000. External quality mechanisms for

health care: summary on the expert project on

visitatie, accreditation, EFQM, and ISO assessment

in European Union Countries. Int J Qual Health

Care 12: 165–169.

Shortell SM, O’Brien JL, Carman JM. 1995. Assessing

the impact of continuous quality improvement/total

quality management: concepts versus implementation.

Health Serv Res 30: 377–401.

Sinha M. 1999. Gaining perspectives: the future of TQM

in public sectors. TQM Mag 11(6): 414–419.

Swiss J. 1992. Adopting TQM to government. Publ

Admin Rev 52(4): 352–356.

Tari JJ, Molina JF, Castejon JL. 2007. The relationship

between quality management practices and their

effects on quality outcomes. Eur J Oper Res 183:

483–501.

Taylor WA, Wright GH. 2003. A longitudinal study of

TQM implementation: factors influencing success and

failure. Omega 31: 97–111.

Int J Health Plann Mgmt (2014)

DOI: 10.1002/hpm

TQMMODEL FOR DRUG REGULATORY AUTHORITIES

United States Pharmacopeial Convention. 2005. Ensuring

the Quality of Medicines in Low-Income Countries, an

Operational Guide. United States Pharmacopeial

Convention: Rockville.

UstunerY,Coskun S. 2004.Qualitymanagement in theTurkish

public sector: a survey. Publ Admin Dev 24(2): 157–171.

Van Hoof T, Bisognano M, Reinertsen J, Meehan T.

2012. Leading quality improvement in primary care:

Copyright © 2014 John Wiley & Sons, Ltd.

recommendations for success. Am J Med 125(9):

869–872.

Wagner C, De Bakker DH, Groenewegen P. 1999. A

measuring instrument for evaluation of quality systems.

Int J Qual Health Care 11: 119–130.

Wagner C, Gulacsi L, Takacs E. 2006. The implementation

of quality management systems in hospitals: a compari-

son between three countries. BMCHealth Serv Res 6: 50.

APPENDIX A

I GENERAL INFORMATION

1. Full name of institution2. Your agency is responsible for regulation of:

a) human medicinesb) veterinary medicinesc) medical devicesd) other

3. Size of organization (number of employees)

a) up to 20b) 20–50c) 50–100d) 100–150e) 150–200f) more than 200

4. Is Quality Management System (QMS) implemented?5. If QMS is certified, please state the date of the first certification.6. In case other management system is certified (ISO 14001, ISO 18001,

ISO 17025 etc.), please indicate which?

II QMS IMPLEMENTATION STRATEGY—REASONS FOR IMPLEMEN-TATION, BENEFITS, CHALLENGES

1. What are the reasons for QMS implementation? ( you can mark morethan one answer)

g) improvement of performance qualityh) this was a demand from our stakeholdersi) to become respectable regulatory agencyj) ISO 9001 is a good management tool for continuous improvementk) operational costs reductionl) other:

2. What are the benefits from QMS implementation in your institution?( you can mark more than one answer)

j) better management controlk) better understanding of problems related to internal communication

Int J Health Plann Mgmt (2014)

DOI: 10.1002/hpm

G. PEJOVIĆ ET AL.

l) better understanding of problems related to internal proceduresm) improvement of communication with stakeholdersn) efficiency improvemento) assessment performance improvementp) operational costs reducedq) it’s useful in new employees educationr) other:

3. Most common challenges met during ISO 9001 implementation ( youcan mark more than one answer):

a) implementation cost are highb) time needed for implementation is very longc) too much paperd) ISO standard requirements are not cleare) it’s additional burden to regular workf) other:

4. The following section is related to your personal/subjective attitude andbeliefs about QMS in your institution. Your opinion can be expressedas: 1 – absolutely not true, 2 – partially true, 3 – true, 4 – more thantrue, 5 – absolutely true.

a) You are satisfied with the way the QMS operates in your company 1 2 3 4 5b) QMS is a management control tool 1 2 3 4 5c) QMS is a driver of innovation 1 2 3 4 5d) QMS plays a strategic role in my organization 1 2 3 4 5e) Managers actively communicate a quality commitment to the employees

1 2 3 4 5f) Employees are encouraged to help implement changes in the organiza-

tion 1 2 3 4 5g) Senior executives take employees’ feedback and surveys seriously 1 2 3 4 5h) Management involves the employees in the setting of its objectives and

plans 1 2 3 4 5i) We have clear strategic objectives for our organization 1 2 3 4 5j) Every employee in our organization is clear about our strategic objective

and the action plans to accomplish it 1 2 3 4 5k) There is bottom-up, top-down and horizontal communication among all

the staff 1 2 3 4 5l) We have a transparent and effective appraisal system for recognizing and

rewarding employees for their efforts 1 2 3 4 5m) We provide training for our employees to improve their competency 1 2 3 4 5n) Our performance quality has been improving steadily 1 2 3 4 5o) We have an effective system to assess our business performance 1 2 3 4 5p) We use information management/information technology to support

quality management (analysis of data regarding business performance,cost and financial aspects in order to support the development ofimprovement priorities) 1 2 3 4 5

Copyright © 2014 John Wiley & Sons, Ltd. Int J Health Plann Mgmt (2014)

DOI: 10.1002/hpm

TQMMODEL FOR DRUG REGULATORY AUTHORITIES

III BENCHMARK OF EUROPEAN MEDICINES AGENCIES (BEMA)

1. Are you familiar with the BEMA methodology?2. Has your agency been included in BEMA process?3. In case your previous answer is ‘yes’, do you consider BEMA method-

ology a useful tool for continual improvement of agency’s performance?4. Please rank the following aspects of BEMA methodology in terms of

their benefit to agency’s performance improvement (based on yourpersonal opinion). Please use the following ranking system, startingfrom 1 (the most beneficial) to 7 (the least beneficial)

a) agency’s self-assessmentb) cooperation with colleagues from other European agenciesc) internal comprehensive preparation and work prior to BEMA visitd) team work and team spirite) it is an opportunity to get a comprehensive picture of agency’s regulatory

performance, as wholef) it is an opportunity to evaluate the level of implemented quality manage-

ment systemg) it gives a possibility for continuous improvement

5. What are the negative aspects of BEMA methodology? (you can markmore than one answer)

a) BEMA questionnaire in some sections is unclearb) not enough time for preparation for BEMA visitc) BEMA assessment and evaluation/ranking procedure was unclear for usd) team work was missing during preparation phasee) agency’s top management didn’t implement all the recommendations for

improvement after BEMA visitf) the importance of BEMAmethodology wasn’t understood among agency’s

staffg) KPI and SPI evaluation system is not described wellh) BEMA team members were not qualified enough for the procedurei) other:

6. Do you support further organization of BEMA visits?

Copyright © 2014 John Wiley & Sons, Ltd. Int J Health Plann Mgmt (2014)

DOI: 10.1002/hpm