The training for health equity network evaluation framework: A pilot study at five health...

116 Education for Health • Volume 27 • Issue 2 (August 2014)

Address for correspondence: Ms. Simone J. Ross, MDR, Lecturer, School of Medicine and Dentistry, James Cook University, Townsville QLD 4811, Australia. E‑mail: [email protected]

The Training for Health Equity Network Evaluation Framework: A Pilot Study at Five Health

Professional SchoolsSimone J. Ross1,6, Robyn Preston1, Iris C. Lindemann2, Marie C. Matte3, Rex Samson4,

Filedito D. Tandinco5, Sarah L. Larkins1, Bjorg Palsdottir6, Andre-Jacques Neusy6

1School of Medicine and Dentistry, Anton Breinl Research Centre for Health Systems Strengthening, James Cook University, 2Flinders University School of Medicine, Australia, 3Division of Medical Education, Dalhousie University, Canada (Formerly of Northern Ontario School of Medicine,

Canada), 4Ateneo de Zamboanga School of Medicine, 5University of the Philippines Manila - School of Health Sciences, Philippines, 6Training for Health Equity Network, Belgium

ABSTRACTBackground: The Training for Health Equity Network (THEnet), a group of diverse health professional schools aspiring toward social accountability, developed and pilot tested a comprehensive evaluation framework to assess progress toward socially accountable health professions education. The evaluation framework provides criteria for schools to assess their level of social accountability within their organization and planning; education, research and service delivery; and the direct and indirect impacts of the school and its graduates, on the community and health system. This paper describes the pilot implementation of testing the evaluation framework across five THEnet schools, and examines whether the evaluation framework was practical and feasible across contexts for the purposes of critical reflection and continuous improvement in terms of progress towards social accountability. Methods: In this pilot study, schools utilized the evaluation framework using a mixed method approach of data collection comprising of workshops, qualitative interviews and focus group discussions, document review and collation and analysis of existing quantitative data. Results: The evaluation framework allowed each school to contextually gather evidence on how it was meeting the aspirational goals of social accountability across a range of school activities, and to identify strengths and areas for improvement and development. Discussion: The evaluation framework pilot study demonstrated how social accountability can be assessed through a critically reflective and comprehensive process. As social accountability focuses on the relationship between health professions schools and health system and health population outcomes, each school was able to demonstrate to students, health professionals, governments, accrediting bodies, communities and other stakeholders how current and future health care needs of populations are addressed in terms of education, research, and service learning.

Keywords: Accreditation of medical schools, evaluation framework, health equity, health services research, health policy, social accountability

Background

Calls for Medical Education Reform

A series of publications in the past 10 years have highlighted the importance of reforming health systems and addressing the shortage and maldistribution of health workers to reduce health inequities within and between countries.[1‑4] Health professional education institutions produce the key health system components of human resources/workforce and information/research and therefore have a central role in the

Access this article online

Quick Response Code:Website: www.educationforhealth.net

DOI: 10.4103/1357-6283.143727

Original Research Article

[Downloaded free from http://www.educationforhealth.net on Monday, June 15, 2015, IP: 137.219.81.143]

Ross, et al.: THEnet evaluation framework pilot study

Education for Health • Volume 27 • Issue 2 (August 2014) 117

complex process of reducing health inequities. The changing global landscape of health and health services has prompted a multitude of national and international groups and bodies to call for reforms in the education of health professionals to meet the changing needs of the 21st Century,[5‑11] including a greater emphasis on social accountability in the accreditation of medical schools.[6,12]

In 1995, the World Health Organization (WHO) defined social accountability as: “the obligation to direct their education, research and service

activities towards addressing the priority health concerns of the community, region, and/or nation they have a mandate to serve. The priority health concerns are to be identified jointly by governments, health care organizations, health professionals and the public.”[13]

There is limited robust evidence on the effectiveness of any medical education reform, including social accountability efforts.[6] To provide this evidence, evaluation tools are required to explore how socially accountable medical or health professional education programs are actualized in different contexts and to inform evidence‑based health and education policies.

In 2008 the Training for Health Equity Network (THEnet), a newly established consortium of health professions schools striving toward social accountability, recognized the need to collaborate to systematically build a common evidence and knowledge base on social accountability in health professions education schools.[13]

The THEnet grew out of a project initiated by the Global Health Education Consortium in 2007‑8, which identified innovative schools of medicine and health sciences already addressing the health and social needs of underserved and marginalized populations.[14] All partner schools [Table 1] located in high, middle and low income countries have an explicit social accountability mandate to train health professionals for service in underserved areas in order to address workforce shortages in rural, isolated and poor urban communities.[13]

Despite contextual variation and settings, all schools aspire to several core educational and social principles [Table 2]. The schools all recruit students from communities with the greatest health care needs and employ preceptors or tutors from the community. Learning occurs in areas of greatest health care need, particularly in community‑based settings. Significantly, all schools have a shared understanding of social accountability.

At THEnet’s first meeting in Havana, Cuba in 2008, it was agreed to amend the WHO’s definition of social accountability by highlighting a focus on the underserved; defined as

communities that have least opportunity to access health services and health professionals: “… The priority health concerns are to be identified jointly by

governments, health care organizations, health professionals and the public (and especially the underserved).”

In response to the need for more practical validated measurement tools, THEnet’s first project was to develop an evaluation framework for schools to assess their progress toward social accountability and thereby their ability to influence health outcomes and health services. Collaboratively, over a period of 2 years (2009‑2010), six of our foundation Schools ‑ Ateneo de Zamboanga University, School of Medicine in the Philippines (ADZU); Flinders University, School of Medicine in Australia (FLINDERS); James Cook University, School of Medicine in Australia, (JCU); Northern Ontario, School of Medicine in Canada (NOSM); University of Philippines in Manila, School of Health Sciences at Leyte, (UPM‑SHS); and Walter Sisulu University, Faculty of Health Sciences in South Africa (WSU) ‑ jointly developed the

Table 1: The THEnet schools

Ateneo de Zamboanga University, School of Medicine in the Philippines (ADZU)*>Comprehensive Community Physician Training Program in Venezuela*Flinders University, School of Medicine in Australia (FLINDERS)*>Gezira University, Faculty of Medicine in Sudan (GEZIRA)>Ghent University, Faculty of Medicine and Health Sciences in Belgium (GHENT)>James Cook University, School of Medicine in Australia (JCU)*>Latin American School of Medicine in Cuba (ELAM)*>Northern Ontario School of Medicine in Canada (NOSM)*>Patan Academy of Health Sciences in Nepal (PAHS)>University of New Mexico Health Sciences Centre in New Mexico (UNM)>University of the Philippines Manila, School of Health Sciences at Leyte in the Philippines (UPM‑SHS)*>Walter Sisulu University, Faculty of Health Sciences, in South Africa (WSU)*>

*Founding Schools; >Current Schools

Table 2: Descriptions of core educational and social principles of THEnet socially accountable health professions schools

Health and social needs of targeted communities guide education, research and service programsStudents recruited from the communities with the greatest health care needsPrograms are located within or in close proximity to the communities they serveMuch of the learning takes place in the community instead of predominantly in university and tertiary hospital settingsCurriculum integrates basic and clinical sciences with population health and social sciences; and early clinical contact increases the relevance and value of theoretical learningPedagogical methodologies are: student, patient and population centered; service‑based and assisted by information communication technologyCommunity‑based practitioners are recruited and trained as teachers and mentorsEmbedded in the health system partnering with health system actors to produce locally relevant competenciesFaculty and programs emphasize and model commitment to public serviceWhole school approach, across all departments, and commitment from the leadership




THEnet’s Evaluation Framework for Socially Accountable Health Professional Education Version 1.0. (EF).[5] Originally produced in 2011 and later published in Medical Teacher, Larkins et al. describes the collaborative development of the evaluation framework.[15]

This paper describes the pilot implementation and findings of testing the evaluation framework across five THEnet schools. The pilot implementation sought to examine whether the evaluation framework was: (1) practical and feasible across contexts; (2) useful for schools for critical reflection on their performance and progress towards greater social Accountability; and (3) useful to assist schools establishing priority areas for research and improvement. The pilot test also sought to examine challenges to implementing the framework in different contexts.

THEnet’s Evaluation Framework for Socially Accountable Health Professional Education

THEnet’s Evaluation Framework for Socially Accountable Health Professional Education[15] is a comprehensive set of processes, measures and tools that identify the key factors affecting a school’s ability to positively influence health outcomes and health systems. THEnet used Boelen and Woollard’s Social Accountability Conceptualization – Production – Usability model (CPU model) as a foundation for the evaluation framework.[16] The CPU model identifies three interdependent domains: (1) Conceptualization of desired professional – ‘collaboration of the kind of professional needed and the system using his/her skills’; (2) Production of desired professional – ‘components of training and learning’ and (3) Usability of professional – ‘initiatives taken to ensure graduates are put to their highest and best use” [15, page 890].

The aim was to develop a practical and useable framework, which could be used by non‑expert evaluators and those whose primary language was not English. Technical concepts in the CPU model were adapted for a larger audience through use of simple English. Conceptualization, became Section One: ‘How does our school work?’; Production became Section Two: ‘What do we do?’; and Utilization became Section Three: ‘What difference do we make?’[15]

Section One of the evaluation framework (How does our school work?) addresses important aspects of the organization and planning of the school frequently neglected in existing evaluation and accreditation frameworks.[15] These include an assessment of values, governance and decision‑making processes and partnerships with the health sector, community groups and policy makers. This section also includes documentation and understanding of the reference population that the school serves, with particular focus on underserved groups within this. Identifying reference populations and health system, is a

prerequisite for evaluating impact of particular strategies and programs.[15]

Section Two (What do we do?) centers on the three standard pillars of what medical/health professions schools do: education, research and service delivery. It looks at features such as the recruitment of students and educators, curriculum, learning methodologies, research, service, and resource allocation. This section aligns with the accreditation criteria for many health professions.[15] Uniquely, the evaluation framework emphasizes how these features link with priority health and health service needs of the schools’ reference populations.

Section Three (What difference do we make?) focuses on the direct and indirect impacts that medical schools and their graduates have on the health of their reference populations and the health system they serve. It includes an assessment of a school’s graduate outcomes (location, discipline and practice of graduates), its engagement and impact on health services and community health and social outcomes and influence on policy makers and other schools.[15]

It is important to note the evaluation framework is not designed as a summative pass or fail exercise, but rather as a process to guide and support schools to take a critical look at their performance, progress, knowledge, skills and capacity in socially accountable health professions education. In addition, the evaluation framework can assist schools to establish priority areas for improvements in education, service and research.

Methods

While THEnet schools share common principles, the schools that pilot tested the evaluation framework vary in enrolment numbers, training settings, curriculum approach and educational methodologies [Table 3].

The methodology had three phases: 1. A workshop; 2. Focus groups and interviews with collation and analysis of existing data; and 3. Individual school reports. Six of THEnet’s foundation schools completed the workshop (JCU, FLINDERS, NOSM, WSU, ADZU, and UPM‑SHS). Five schools piloted the evaluation framework over a 2‑year period; FLINDERS, JCU and NOSM in 2010 and UPM‑SHS and ADZU in 2011. UPM‑SHS and ADZU piloted a revised version of the evaluation framework from the 2010 pilot. Figure 1 outlines the methodology of the pilot study across schools.

Phase 1: Each pilot test began with a workshop with faculty, staff, students and community members to assess the evaluation framework as a critical reflection tool within the context of each school.




Phase 2: Progress against the aspirations in each section of the evaluation framework was assessed through interviews and/or focus groups with academic/faculty and professional staff members, students at different levels of the course, and community members and health sector representatives.

An implementation guide was created to standardize the processes of administering the framework across schools. It was agreed early that each school was to conduct focus groups and interviews with students, faculty and staff, and community members to receive a full range of perspectives of social

Table 3: THEnet schools involved in testing the EF

School Enrolment and graduate numbers (2012)

Training setting Curriculum approach/educational methodologies

Ateneo de Zamboanga University, School of Medicine, Philippines

Established: 1994

Four year post‑graduate program

152 students in levels 1 to 4221 graduates

Close to 50% of the 4‑year program is based in the community. The fourth year is entirely spent in the community implementing a 3‑year Comprehensive Health Plan for health development through inter‑sectoral approaches. Likewise, each student implements an interventional research project. This latter component in their training is meant to help provide a database for the community and likewise intervene in resolving an existing health problem in the community through research

ADZU curriculum involves learning strategies structured along the concept of solving clinical problems (problem‑based). It further involves the teaching of medicine around functions and competences required to practice medicine in a specific setting (competency‑based). As early as first year, students are exposed to patients both in clinics and communities where the focus on the practice of medicine as applied to a group or population is emphasized (community‑based). The would‑be‑graduate is to be a competent professional who can practice medicine at a level of defined proficiency in accord with national health needs

Flinders University, South Australia

Established: 1974


563 students in years 1‑4835 graduates since graduate entry program commenced in 1996

Flinders University trains and supports health care providers in rural communities as clinical faculty. Options for students in gaining clinical experience occurs in varied settings‑from traditional rotations in big city hospitals, through single rotations or year‑long involvement in health care delivery in outer urban, rural or remote communities, to electives almost anywhere in the world. More than half of the students in third year select from a growing array of clinical experiences in rural and remote communities

The Flinders University School of Medicine was one of the first in Australia to develop a 4‑year graduate‑entry medical program as an alternative to traditional 5‑ to 6‑year courses for school leavers. The curriculum is fully integrated and linked with local health services. There is an emphasis on problem based learning in the first 2 years. In the final 2 years students can choose from a range of clinical experience options many of which require students to relocate to rural and remote community settings. Flinders has recently introduced a school leaver entry pathway and an Indigenous entry stream, and from 2011 applicants can apply to undertake the Medicine program either in Adelaide, or in Darwin through the Flinders University Northern Territory Medical Program

James Cook University, northern Queensland, Australia

Established: 1999

Six year undergraduate program

1005 students550 graduates (2000‑12)

The School of Medicine at James Cook University offers a 6‑year full time undergraduate degree in medicine and surgery. Students experience small group teaching, modern facilities and early clinical practice. Students have the opportunity to take international electives (there are scholarships and bursaries available to travel overseas) and develop specialist knowledge in the areas of rural, remote, tropical and Indigenous health during at least 20 weeks of small rural or remote placements throughout Australia

The course is based at the Townsville campus for the first 3 years, with an overlapping wedge model of basic science and clinical skills training. The majority of students in years 4, 5 and 6 relocate for the entire 2 years to other School of Medicine teaching sites such as Cairns, Mackay and Darwin. In years 5 and 6, students spend most of their learning time in hospitals and primary health care services. Curriculum has a strong focus on the health of rural, remote, Indigenous and tropical communities

Northern Ontario School of Medicine, Canada

Established: 2005


64 students per year (how many years? 1‑4) or a total of 256 in the program170 residents

NOSM developed and delivers a highly innovative e‑curriculum and distributed community engaged education program. The clinical learning of students takes place in more than 70 indigenous and remote communities of northern Canada and includes year‑long Comprehensive Community Clerkship

NOSM employs a case‑based approach to focus student learning. The emphasis is on self‑directed learning and the role of faculty tutors is to facilitate learning. In years 1 and 2, this model is employed with the Case Based Learning Sessions and the Topic Oriented Sessions. In year 3, Virtual Academic rounds are utilized to bring small groups of learners together to discuss cases from their personal clinical experiences

University of the Philippines, School of Health Science, Philippines

Established: 1976

Unique step ladder curriculum with service components

189 students enrolled (86 in midwifery, 35 in nursing, and 68 in medicine)Since (date) 125 graduates from the medical program

Unlike the normal admissions system in the University, UPM‑SHS students are selected by the barangay or village on the premise that people who are actually members of socio‑economically deprived communities will have greater commitment and, therefore, are more likely to return to serve their own underserved areas. Upon nomination, the selected scholar publicly pledges to return to the community to render service as a health worker. This pledge operates as a type of “social contract” entered into by the nominee and his or her barangay. In turn, the barangay pledges to provide moral and material support to the student while s/he trains at UPM‑SHS. To date, this “social contract” has evolved into the written “return service agreement”

The UPM‑SHS program in Leyte has a competency‑based and community‑based stepladder curriculum, involving the training of a broad range of health manpower‑from midwife to medical doctor‑in a single, sequential and continuous curriculum. Training covers five competency areas threaded through all levels, which include those as: (a) health care provider, (b) community mobilizer/organizer, (c) health service manager, (d) trainer/educator and (e) researcher. Within this system, a student is trained first as a midwife, then progresses to becoming a nurse with a BS Nursing degree, and eventually, becomes a doctor of medicine. Fifty percent of their training is spent in the community. In between program levels, students return to serve their communities (service leave) before progressing to the next level




accountability across the school. The number of focus groups and interviews and exactly who participated was left up to each school to define within their own context and educational system. All schools had the agreement to purposefully choose participants who would be open with critical reflection, and/or with corporate knowledge of the school. Participation was voluntary and steps were taken at each school to assure participant confidentiality and anonymity.

The range of focus groups and interviews conducted depended on the context of each school. For example, ADZU has a 4‑year postgraduate course, and conducted seven semi‑structured focus groups with students (five) and community representatives (two), and seven interviews with faculty/staff. In comparison, JCU has a 6‑year undergraduate course, and conducted seven semi‑structured focus groups with students (two), faculty/staff (three), community representatives (one) and volunteer patients (one).

In comparison, FLINDERS and UPM‑SHS piloted the evaluation framework across a school that delivers a number of different health professions courses; therefore FLINDERS conducted 10 semi‑structured focus groups with students (four), faculty (five) and stakeholders (one), and 21 interviews with stakeholders (2) and faculty/staff (nineteen). In comparison, UPM‑SHS conducted eight semi‑structured focus groups with medical students (two), nursing students (one), faculty (one), alumni working in rural settings (two), public health service providers in a learning community (one) and representatives from the schools reference community (one).

To assure consistency of data collection an interview guide was created. It was first piloted at NOSM, JCU and FLINDERS, and then at UPM‑SHS and ADZU. This ensured all schools asked the same six key questions at each focus group or interview [Table 4].

Both ADZU and UPM‑SHS schools translated the questions into local dialects and retranslated participants’ input into English. The data at all schools was analyzed using a qualitative ‘grounded theory’ approach, developing theories and theoretical propositions from the data.[17] JCU and FLINDERS also coded electronically, JCU coded using Microsoft Excel and FLINDERS using NVivo. No information that might identify individuals was recorded at any school.

Progress toward the aspirations was further analyzed through collation of existing sources of evidence as suggested in the evaluation framework. A full description of the development, and the evaluation framework can be found in Larkins et al.[15]

Phase 3: Each school reported on the results of their own pilot study and the evaluation framework was refined and finalized in accordance with the findings. The report included: (1) feedback on the evaluation framework and its feasibility for implementation, (2) feedback on the pilot process and suggestions

Table 4: Interview/focus group guide

Each school asked the following six questionsWhat is your understanding of the values of the school? What does the school believe in?Can you demonstrate how these values are shown in the day to day workings of the School?What groups/populations/communities do you believe the School focuses upon? To what extent do you perceive the educational programs address these communities’ needs?Are you aware of any means by which you can contribute to the decision making processes of the school – in relation to curriculum development, research agendas and community partnerships?What has been your experience in teaching in an interprofessional environment? Are you aware of interprofessional learning opportunities for students at the undergraduate and postgraduate levels of your school?Do you believe the school has allocated sufficient resources to all stakeholders involved in the planning and delivery of the medical education programs to meet their needs? If yes, can you provide examples? If no, can you provide examples?

Figure 1: Process for development and pilot test of the evaluation framework




for adaptations to the process for broader implementation, and (3) key recommendations for improvement for each school in response to the results of the pilot implementation.

Findings from the five schools were collated and the results were discussed to confirm findings and to consider contextual variations between schools, which may have impacted on implementation.

Each school individually obtained ethics approval to conduct the pilot implementation.

Results

Each school found that the evaluation framework was applicable in their cultural and school contexts and that information could be collated for most of the key components. The evaluation framework was used in its entirety and as a ‘whole of school’ process, not at department or program levels. Full support from school leaders, including deans, was essential to both effectively conduct the pilot studies and implement the study’s findings.

Each school agreed that the evaluation framework provides an opportunity to take a comprehensive and critical look at their performance and progress toward greater social accountability. The three sections of the evaluation framework identified the areas of strength, gaps and a list of priority areas for research that will guide each school to translate social accountability in practice and to develop and establish priority areas for research and program development. The key findings from the pilot evaluation framework implementation are summarized below and in Table 5.

Section One: How Does Our School Work?

This section produced similar findings across schools and posed a set of challenges. For example, at some schools, although focus group or interview participants could identify a reference population, there was limited formal documentation to support this understanding. Also, all schools include underserved communities in their reference population; however, the term “under‑served” is highly dependent on the context and needs to be better defined by local stakeholders.

Although the types of partnerships and collaborations varied across schools, each could verify through meeting notes, focus group discussions and memoranda of understanding that government and non‑government organizations, communities and community health centers, local and international institutions, and student groups are stakeholders involved in decision‑making processes within the school.

All schools also identified that finding documentation in support of Section One was difficult, as some of the

decision‑making processes were not well documented, could not be located or were held only in the corporate memories of key school personnel. This was marked as a recommendation or gap in each school report. However, this result was also found to be a benefit to schools, as documents are now being written which allow clearer communication of social accountability activities to new faculty and staff joining the school.

Section Two: What Do We Do?

This section identified many similar findings across schools, despite each school having different strategies for meeting their own social accountability goals. Student support was contextualized to student educational needs, and students were well supported at all schools. All schools could cite examples of collaborative and community based research projects in all areas of health and wellbeing. However, only some schools had sustainable funding for research with underserved communities. Each school has students working in the community, which provides a positive effect on health service delivery.

Section 3: What Difference Do We Make?

While the aspirations in section three are largely overlooked in most accreditation measures and are frequently considered beyond the scope of health professional education institutions, these factors are crucial components when evaluating social accountability in health professions education. The three schools (JCU, ADZU, UPM‑SHS) that were actively tracking graduates were able to demonstrate higher retention rates of graduates than traditional schools within priority areas, where health professionals are most needed. Different strategies were used to determine graduate impact beyond graduation. Many of the schools disseminated their social accountability research and community based practices in different forums including local, national, and international conferences. The UPM‑SHS and ADZU schools both have had positive impact on partnering with other non‑THEnet medical schools in their own country and elsewhere to develop their socially accountable medical program.

Nonetheless, schools found that fully completing Section 3 would require significantly more resources, tools and time than allocated to this project.

Discussion

To ensure evidence‑based reform as advocated by the Lancet Commission,[6] health professional schools must be able to assess their effectiveness in addressing the needs of the communities and health systems in which they operate. This includes an evaluation and long‑term impact study on where their graduates are working, in what type of practice they are engaged, and whether the competencies they obtained




Table 5: Summary table of findings from schools who conducted the evaluation framework pilot study

Section Key component JCU Flinders NOSM UPM‑SHS ADZUHow does our school work?

What do we believe in?(Values)

Participants were aware of the school’s values and stated strategic intent (focused on rural, remote, Indigenous and underserved populations) and identified the values by example rather than name

Awareness of the school’s values varied across participants, however, most were able to identify where values supporting socially accountable practices were demonstrated within school activities

All participants were aware of the school social accountability mandate. It is well advertised.The School is established as not for profit

Social accountability values are known and understood by the faculty, students and alumni, although not all community representatives included in the interview knew the school’s values

Participants knew about the school’s values and were able to give examples on how the school operationalized these valuesThe school is not for profit

Who do we serve?(Reference Population and Health System)

The reference population was easily recognized by all participants, although no formal statement of the reference population, ‘JCU patch’ in northern Queensland could be located

No clear reference populations was identified in documentation or considered by participants to be ‘our populations’

The school recognizes and defines the populations they serve.Participants identified the educational program serves diverse populations

The school has a clear rationale for identification of populations and underserved populations are well defined and emphasized

The school has available documents outlining the school’s population.>80% of people lack in medical attention in the reference population

How do we work with others?(Engagement)

Participants identified varied partnerships and relationships in decision making at the school but wanted more feedback on contributions

Participants felt partnerships could be better informed, recognized and nurtured

Participants identified many avenues to be involved in the school decision making processes, but identified infrastructure to support research activities could be better developed

Varied partnerships are evident. Students also work with the community in planning, implementing and evaluating health and other community programs

Varied partnerships are evident. Some participants felt they did not have a strong lobbying power and opportunity to influence the school’s operation, this has since been rectified

How do we make decisions?(Governance)

There is corporate memory but limited documentation of community participation and feedback. The corporate knowledge appears to reside in the stakeholders, not necessarily documented (in KC1.3)

Health community included strongly in governance and curricula decisions‑some nice examples of lay community inclusion but only few. Much decision making not documented (Sealed with a handshake!)

Governance and curricula decisions are collaborative in nature and involve the schools stakeholders

There seems to be no evidence of direct community participation in the school’s corporate, fiscal and academic governance. Selection of scholars, is the primary responsibility of the communities providing moral and financial support to their students

The greatest challenge is the poor documentation practice in the school. Some processes or protocols currently being implemented exist only in the corporate memory of the faculty and still needs to be penned down

What do we do?

Where do we invest resources?(Field operations)

The school regularly applies for and wins grants for research, that supports activities with underserved communitiesParticipants advised resources are ‘great’ but are concerned about increased student numbers

School is strong in attracting funding for innovation and expansionMany participants feel resources are stretched‑but what is ‘sufficient resources’? No evidence that students success differs across sites

Participants identified the evidence of the allocation of sufficient resources could be seen in the success of studentsThere was a perception that dollars allocated to research initiatives on campus and in the community have fallen off

Resources for community engagement and program operationalization are distributed according to priority needs of the community, and communities support student allowancesFunding is primarily sourced from government

Source of funding for the school is primarily from the tuition fees of students and occasional donation from donor organizationsThe school actively supports students in soliciting resources from Local Government for community projects

What, where and how do we teach?(Education program)

Participants requested more inter‑professional learning in the curriculumMany curriculum documents and policies were located and reflected a socially accountable curriculum/valuesStudent support mechanisms are vast

Interprofessional education was lackingParticipants generally felt the curriculum reflected priority health and social needs of the communitiesStudent participants felt the school did not prioritize and therefore value social aspects of health in the curriculum

Participants identified the interprofessional program was not well defined or communicated, and for some non‑existentStudent support includes personal and financial services

Interprofessional education opportunities are available to all students through the competency‑based and community‑based step‑ladder curriculum, which integrates the training of midwives, nurses and doctors into a single, continuous curriculum

The curriculum of the school was developed through a consultative process with the various stakeholders in the region.For the community teaching sites, the school now preferably selects communities where the Municipal Officer is an alumni of the school

Who will we teach?(Learners)

Student Recruitment is from our reference population of northern Australia, which meets our strategic intentIn 2010 we had low attrition rates of 3.6% in years 1‑3,

No targets for admission/selection of any defined populations is set, however, there are a number of pathways which favor rural and remote

Student Recruitment is from our reference population of Northern OntarioThere is a very low attrition numbers with 2

The students from our reference population is 100%, all come from underserved areasThe school also has lateral entrants who have

A community representative is included in the selection panel screening of students for admission and scholarship. 30% of

Contd...




Table 5: Contd...

Section Key component JCU Flinders NOSM UPM‑SHS ADZUwith no attrition in years 4‑6 applicants. Participants

were highly supportive of such pathways

students withdrawing in 5 years (0.6%)

graduated from other colleges and enter the school as first years

students are scholars (scholarship holders)The school has a total of 152 enrolled students

Who does the teaching?(Educators)

Educators from diverse professional and cultural backgroundsCommunity preceptors are recognized with adjunct titles, including indigenous cultural mentors and ‘on the day’ simulated volunteer patients who provide training to our students

Strong recent increases in indigenous academicsCommunity preceptors and local support persons in all community placement sitesIdentified a need for staff to all attend cultural awareness programs

We engage and support community and community health service providers as educators, strengthening local health servicesStaff reflect the demographics of our reference population

Faculty/staff and promotions reflect a diverse mix of professional, cultural and community backgroundsCommunity preceptors are formalized as ‘adjunct’ appointments, and clinicians are given university appointments as lecturers/clinical preceptors

Educators are from a diverse background as the School has moved toward interprofessional teachingIn community teaching sites, Municipal Health Officers become preceptors and nurses, midwives, and barangay (community) health workers supervise students on practical placements

How does our research program relate to our missions and values?(Research)

Our research priority agenda is the same as our strategic intentThere are a number of collaborative research projects being undertaken in and with the community, focusing on priority health needs

No defined research agenda for the whole school. Participants felt a defined research agenda was limiting especially for some staffThere are many examples to be found across the school of socially accountable research

The school supports a research agenda that includes priority health needs of our reference populationResearch projects are community based involving community stakeholders and students

The school’s research agenda reflects social accountability and aligns with national and regional priorities.Projects focus on culturally appropriate, affordable and innovative solutions to health service problems and health promotion

The school in partnership with the Zamboanga City Consortium for Health Research and Development spearheaded the creation of the region’s Research Priority Agenda, with the National Unified Health Research Agenda being subscribed to by the school

What contribution do we make to the delivery of health care?(Service)

The school provides the opportunity for clinical staff to deliver locum/relieving in rural communitiesStudents undertake a minimum of 20 weeks of placement in remote and/or small rural towns from year 1

Staff and students are directly involved in service delivery but this is not well documented and evidence is mainly anecdotalStudent participants in rural education programs felt strongly they were contributing to community health care

Educators and students are involved in service delivery that is directly related to changing priority health needs of reference populationsStudent placements in the community occur from year 1, with 3×4 week in years 1‑2, and an 8 months clerkship in year 3

Community participants say students fill the gaps in health care deliveryMedical students spend a total of 1.5 years (6 months clerkship, 12 months internship) in the community. Community exposure begins at the midwifery then nursing levels

Most of the faculty members are part time employees of the medical school and primarily involved in health service deliveryStudents spend 50% of their training time in their community and render services at the local health center and hospitals

What difference do we make?

What are our graduates doing?(Human resources)

51% of first year graduates (interns) from 2005 to 2010 have elected to undertake their internship in ‘outer regional’ and ‘remote’ locations. Half of our graduates come back to North Queensland after their intern year from 2006 to 2008 to practice Medicine

Limited comprehensive data is available regarding graduate outcomes.

The School is seen by its community partners and local government as a key advocate and their link to the populations of Northern Ontario.The current CPD policies and processes do not appear to clearly align with the social accountability of the school

78% of the schools’ graduates are now licensed physicians and almost 90% are still serving in the country; 71% are working in government‑run health facilities; 15% are specialists in various fields and have returned to practice in their area

80% of the 221 graduates are already licensed doctors, 84% of which are in the region working in government‑run health facilities in city health or community hospitals or in training at various hospitalsThrough its telehealth program, graduates are provided a referral system where they can link to faculty for referral of any clinical dilemma

What difference have we made to our reference population and reference health system?

Social, economic and infrastructure support, plus retention of health professionals in the communityHealth information, health, economic and social

School has shown strong leadership in achieving more accessible medical education and promoting health workforce in priority areas, especially in the northern territory

There is currently not enough evidence at this point in the school’s development to provide definitive evidence of achievement

The school has influenced the restructuring of the health care delivery systemThe school provides opportunities for health professional education

At the community level, students of the school conduct community diagnosis to identify their adopted community’s health needs and implement interventions

Contd...




during their education adequately prepared them to serve their communities, including the underserved. The evaluation framework underscores these impacts and engages institutions in systems thinking that is necessary to understand and address the complex challenge of improving education to strengthen health system. It provides a comprehensive framework to gather and analyze information that can help align education and health systems planning.[6,18]

Although operating in very different contexts in low and high resource countries, THEnet schools found the evaluation framework a practical and useful tool to assess their progress toward greater social accountability. While all of the schools have missions and activities already aligned with the core educational and social principles of social accountability, each school was able to identify areas for improvement and further

research. The implementation of the evaluation framework provided an opportunity for schools to demonstrate their commitment to social accountability through critical self‑reflection. It highlighted to each school the need for resources to further their work and also the importance of producing demonstrable outcomes or impact, for example, that graduates are serving in communities of need and contribute to improving health outcomes of the population they serve and to strengthening the local health system.

In terms of pilot implementation of the evaluation framework, differences between more financially resourced and less financially resourced schools were not prominent. The major differences in the results related to the diverse operational styles and context of the schools rather than any financial, cultural or social variations. This indicates that the evaluation

Table 5: Contd...

Section Key component JCU Flinders NOSM UPM‑SHS ADZU(Partnerships and effects on health outcomes)

outcomes indicators are regularly compiled by students‑analysis is required

to underprivileged, low‑income familiesHealth information, health, economic and social outcomes indicators are regularly compiled by students‑analysis is required

to address theseImpact studies need to be done to assess the school’s effect on the health status of the region, although health priorities such as infant mortality have significantly dropped

How have we shared our ideas and influenced others?(Dissemination, promotion and sustainability)

Conferences, peer‑reviewed publications, and newslettersWorking on influencing policy to produce a more socially accountable and responsive health workforce

The school aims to disseminate where possible through conference presentations, publications and similarThe school encourages mutual visits and relationship building with schools interested in social accountability

There is little evidence at this point in the school’s development to provide definitive evidence that we have influenced policymakers, education providers and other stakeholders to transform the health system

The experience of the school influenced the development and adoption of the Return Service Agreement of the entire UP Manila for all its graduates of the health professionsUPM‑SHS and its community‑based strategies have been presented in various local and international forums

Our unique curriculum and high retention rate of graduates to practice locally received much attention especially from institutions outside the PhilippinesWe will head a technical panel to review the obsolete Philippine Medical Act of 1956, which has great potential to change the way medicine will be practiced and taught

What impact have we made with other schools?(Peer support and replication)

Visits with THEnet partner schools, allowing for shared understanding of school contextShare resources with low income THEnet partner schools (i.e., text books, library resources)

The school has shared its curriculum with other schools, emphasizing a community‑based model of educationSchool prioritizes its work with THEnet partner schools to share resources/ideas and to develop resources, which can be used by others.

School has been replicated in two other regions of the country by the university itself and in two other state universitiesSchool has hosted trainings for PHC experts, participants of which have been fielded to 26 developing countries.Input to development of other health and non‑health ladder curriculum in the Philippines (associate in health science education and other professions)

The school of medicine was tapped by the National University of Laos in June 2005, to help conceptualize their medical and family medicine programThe school also visited and was consulted by the Pattan Academy of Health Sciences in Nepal in 2008 for their Medical program.Currently, one of the faculty members was hired as consultant to a newly established medical school in Malaysia




framework is applicable across contexts and is flexible enough to cater for differing cultures and resource levels of schools.

Context‑sensitive instruments such as the evaluation framework have the potential to increase our understanding of factors that hinder or facilitate health equity. Collectively, contextual differences between THEnet schools greatly assisted the evaluation framework development.

Further Research

This pilot study has highlighted the need for more research as well as better tools and resources to assess the links between how academic institutions operate, their programs and activities, and ultimately how these factors impact their graduates, reference population and the health system in which they operate.

Recently there have been initiatives to develop tools that track graduate outcomes; most notably the Medical School Outcome Database in Australia and New Zealand.[19] THEnet is building on those efforts and that of individual member schools to develop graduate outcome tracking tools for the evaluation framework that can be used across contexts and will provide a larger cohort and allow cross‑institutional comparison in a variety of contexts and continents increasing reliability and validity. It is also designing impact evaluation tools to further strengthen the resources available as part of the evaluation framework.

THEnet is evolving into a broader learning community and since the pilot study, five new schools have joined THEnet. At the time of writing this paper (January 2013) each of these schools is in various stages of implementing the evaluation framework, thus providing additional feedback. This feedback, as well as new tools and research findings will be integrated into version 2.0 of the evaluation framework. This additional work will improve the understanding of the links between the three sections in the evaluation framework and its core components and thereby strengthen the evaluation framework.

Ultimately, the evaluation framework will be used to gather data across schools and contexts with the aim of using the results for evidence‑based policy guidance and continuous, transformational improvement and reform of health professions education at national, local, institutional and practice levels.

Limitations

There were some limitations in this pilot study mostly due to limited financial and time resources. First, in all piloting schools, faculty conducted data collection, focus group discussions, interviews and reporting; there were no external evaluators. We feel this is not problematic for this pilot study, as it was conducted to look at the feasibility and practicality of the evaluation framework; however, internal rather

than external researchers presented opportunity for bias. Nonetheless internal evaluators offered strength in terms of relevant understanding, and all researchers were experienced in qualitative methods and used reflexivity and researcher triangulation to minimize potential conflict.

Second, the schools that created the evaluation framework also evaluated its feasibility in the pilot study. One recommendation of the pilot is to further develop the evaluation framework to be used as both a critically reflective tool and a peer review tool, with peer review teams to address any potential bias.

Third, faculty who not only designed the evaluation framework but who held an important position at the school conducted focus groups and interviews. This was identified as a potential conflict early and the two Australian schools (JCU and FLINDERS) organized non‑THEnet member facilitators for the interactions with participants.

Last, there were translation problems from English to Filipino for the two Philippine schools (UPM‑SHS and ADZU), as the Filipino language does not have a term for ‘Social Accountability’. While the evaluation framework has been translated into Spanish and French; common terms and understandings for social accountability need to be developed in languages other than English to add real meaning.

Conclusions

This relatively small but diverse pilot study has demonstrated that the evaluation framework is a practical and useful tool, and whole of school reflective process for health professional schools to assess their progress toward social accountability. The evaluation framework is a feedback mechanism for a school, to improve its programs and activities in terms of relevance, equity, and quality and to further develop the school’s partnership with the health system.

This pilot study has informed THEnet’s research activities, which will ultimately contribute to strengthening the evaluation framework. Additional research and tools are needed to improve the evaluation framework as well as further testing at schools engaged or interested in reforming health professions education. It is an evolving tool that will continue to develop by our group as more schools within and beyond THEnet provide feedback. These new tools and the evidence gathered using the evaluation framework will enable schools in different cultural and social settings to demonstrate to students, health professionals, governments, funders and communities that they are addressing the current and future health care needs of their local populations.

The evaluation framework is available from THEnet website (www.thenetcommunity.org) in English, French and Spanish,




or can be electronically provided through email consultation at [email protected]. An interactive on‑line version will be available at the above website from mid‑2014.

AcknowledgmentsThe authors also wish to acknowledge other contributors to the evaluation framework: Charles Boelen, Kate Brennan, Juan Carrizo, Pasqualito Concepcion, Fortunato L. Cristobal, Aaron Goldstein, Jennene Greenhill, Dan Hunt, Jehu Iputo, Afdal Kunting, Joel Lanphear, David Marsh, Khaya Mfenyana, Jose Alvin P Mojica, Ileana del Rosario Morales Suárez, Richard Murray, David Prideaux, Jusie Lydia Siega‑Sur, Roger Strasser, Paul Worley, Sarah Strasser, and Zorayda Leopando.

Please note: This paper discusses University of the Philippines Manila – School of Health Sciences at the time of evaluation (2011–2012), but with deep respect for the devastating effects of Typhoon Haiyan that struck Leyte in early November 2013, this paper is dedicated to the faculty and students of UPM‑SHS. THEnet and partner schools are committed to working with UPM‑SHS in their rebuilding efforts. For more information see: www.thenetcommunity.org

References1. UN Millennium Project, Task Force on Child Health and

Maternal Health. Who’s Got the Power? Transforming health systems for women and children. Available from: http://www.unmillenniumproject.org/reports/tf_health.htm [Last aaceesed on 2012 Feb 02].

2. World Health Organization. The World Health Report 2006: Working Together for Health. Geneva. WHO. 2006. Available from: http://www.who.int/whr/2006/en/index.html. [Last accessed on 2012 Mar 20].

3. World Health Organization. Closing the Gap in a Generation: Health Equity Through Action on the Social Determinants of Health. Geneva.WHO. 2008. Available from: http://whqlibdoc.who.int/hq/2008/WHO_IER_CSDH_08.1_eng.pdf [Last accessed on 2012 Feb 02].

4. Frenk J. The Global Health System: Strengthening National Health Systems as the Next Step for Global Progress. PLoS Med [serial on the Internet]. 2010; 7 (1). Available from http://www.plosmedicine.org/article/info%3Adoi%2F10.1371%2Fjournal.pmed. 1000089 [Last accessed on 2012 Feb 02].

5. BusingN,RosenfieldJ,RourkeJ.TheFutureofMedicalEducationin Canada (FMEC): A Collective Vision for MD Education. 2010. Available from: http://www.afmc.ca/fmec/pdf/collective_vision.pdf [Last accessed on 2012 Dec 21].

6. Frenk J, Chen L, Bhutta ZA, Cohen J, Crisp N, Evans T, et al. Health Professionals for a new Century: Transforming Education to Strengthen Health Systems in an Interdependent World. Lancet 2010;376:1923‑58.

7. General Medical Council. Tomorrow’s Doctors: Outcomes and Standards for Undergraduate Medical Education. 2011. Available from: http://www.gmc‑uk.org/Tomorrow_s_Doctors_1011.pdf_48905759.pdf [Last accessed on 2012 Apr 15].

8. Global Consensus for Social Accountability of Medical Schools. Global Consensus for Social Accountability of Medical Schools. 2011. Position statement. Available from: http://www.healthsocialaccountability.org [Last accessed on 2012 Apr 15].

9. Lueddeke GR. Transforming Medical Education for the 21st Century: Megatrends, Priorities and Change. London: Radcliffe Publishing; 2012. ISBN‑13: 978 184619 969 1.

10. Institute of Medicine. Health Professions Education: A Bridge to Quality. Washington, DC: National Academy Press; 2003. p. 1. Available from: http://www.iom.edu/Reports/2003/Health‑Professions‑Education‑A‑Bridge‑to‑Quality.aspx [Last accessed on 2013 Jan 21].

11. Cooke M, Irby DM, O’Brien BC. A Summary of Educating Physicians: A Call for Reform of Medical School and Residency. Stanford, CA: The Carnegie Foundation for the Advancement of Teaching; 2010. Available from: http://www.carnegiefoundation.org/elibrary/summary‑educating‑physicians [Last accessed on 2012 Apr 05].

12. BoelenC,HeckJ.DefiningandMeasuringtheSocialAccountabilityof Medical Schools 1995; No. WHO/HRH/95.7 Unpublished; Geneva: World Health Organisation. http://www.who.int/iris/handle/10665/59441 [Last accessed on 2012 Feb 12].

13. Palsdottir B, Neusy AJ. Global health: Networking innovative academic institutions. Infect Dis Clin North Am 2011;25:337‑46.

14. Pálsdóttir B, Neusy AJ, Reed G. Building the Evidence Base: Networking innovative socially accountable medical educational programs. Educ Health 2008;21:1‑6.

15. Larkins SL, Preston R, Matte MC, Lindemann IC, Samson R, Tandinco FD, et al. Measuring social accountability in health professional education: Development and international pilot testing of an evaluation framework. Med Teach 2013;35:32‑45.

16. Boelen C, Woollard B. Social accountability and accreditation: A new frontier for educational institutions. Med Educ 2009;43:887‑94.

17. Patton M. Qualitative research and evaluation methods. 2nd ed. Newbury Park, CA: Sage Publications; 1990.

18. de Savigny D, Adams T. Systems thinking for health systems strengthening. Geneva: Alliance for Health Policy and Systems Research. World Health Organization. 2009. Available from: http://www.who.int/alliance‑hpsr/resources/9789241563895/en/index.html [Last accessed on 2012 Dec 18].

19. Gerber JP, Landau LI. Driving change in rural workforce planning: The medical schools outcomes database. Aust J Prim Health 2010;16:36‑9.

How to cite this article: Ross SJ, Preston R, Lindemann IC, Matte MC, Samson R, Tandinco FD, et al. The training for health equity network evaluation framework: A pilot study at five health professional schools. Educ Health 2014;27:116-26.

Source of Support: The work reported in this paper was funded by Atlantic Charities Trust and the support of the Arcadia Foundation received through the Build Project 501(c3.). Conflict of Interest: None declared.


The training for health equity network evaluation framework: A pilot study at five health...

Documents

Transcript of The training for health equity network evaluation framework: A pilot study at five health...