2015, 1–13, Early Online

Predictors of professional behaviour andacademic outcomes in a UK medical school:A longitudinal cohort study

JANE ADAM1, MILES BORE2, ROY CHILDS3, JASON DUNN1, JEAN McKENDREE1, DON MUNRO2 &DAVID POWIS2

1Hull York Medical School, UK, 2University of Newcastle, Australia, 3Team Focus, UK

Abstract

Background: Over the past 70 years, there has been a recurring debate in the literature and in the popular press about how best

to select medical students. This implies that we are still not getting it right: either some students are unsuited to medicine or the

graduating doctors are considered unsatisfactory, or both.

Aim: To determine whether particular variables at the point of selection might distinguish those more likely to become satisfactory

professional doctors, by following a complete intake cohort of students throughout medical school and analysing all the data used

for the students’ selection, their performance on a range of other potential selection tests, academic and clinical assessments

throughout their studies, and records of professional behaviour covering the entire five years of the course.

Methods: A longitudinal database captured the following anonymised information for every student (n¼ 146) admitted in 2007 to

the Hull York Medical School (HYMS) in the UK: demographic data (age, sex, citizenship); performance in each component of the

selection procedure; performance in some other possible selection instruments (cognitive and non-cognitive psychometric tests);

professional behaviour in tutorials and in other clinical settings; academic performance, clinical and communication skills at

summative assessments throughout; professional behaviour lapses monitored routinely as part of the fitness-to-practise

procedures. Correlations were sought between predictor variables and criterion variables chosen to demonstrate the full range of

course outcomes from failure to complete the course to graduation with honours, and to reveal clinical and professional strengths

and weaknesses.

Results: Student demography was found to be an important predictor of outcomes, with females, younger students and British

citizens performing better overall. The selection variable ‘‘HYMS academic score’’, based on prior academic performance, was a

significant predictor of components of Year 4 written and Year 5 clinical examinations. Some cognitive subtest scores from the UK

Clinical Aptitude Test (UKCAT) and the UKCAT total score were also significant predictors of the same components, and a unique

predictor of the Year 5 written examination. A number of the non-cognitive tests were significant independent predictors of Years

4 and 5 clinical performance, and of lapses in professional behaviour. First- and second-year tutor ratings were significant

predictors of all outcomes, both desirable and undesirable. Performance in Years 1 and 2 written exams did not predict

performance in Year 4 but did generally predict Year 5 written and clinical performance.

Conclusions: Measures of a range of relevant selection attributes and personal qualities can predict intermediate and end of

course achievements in academic, clinical and professional behaviour domains. In this study HYMS academic score, some UKCAT

subtest scores and the total UKCAT score, and some non-cognitive tests completed at the outset of studies, together predicted

outcomes most comprehensively. Tutor evaluation of students early in the course also identified the more and less successful

students in the three domains of academic, clinical and professional performance. These results may be helpful in informing the

future development of selection tools.

Introduction

Writing in the British Medical Journal in 1946, Smyth observed

that ‘‘Existing methods of selection [of medical students] which

worked well in the past may no longer be the best possible in

changing conditions’’, further suggesting ‘‘we want . . . two

independent tests or sets of tests – the one for ability, the other

for character’’ (Smyth 1946). Though most medical students do

graduate and become professional and capable doctors, the

subsequent and continuing debate creates the impression that

medical schools are still selecting unsuitable students

(Campbell 1974; Lockhart 1981; Lancet editorial 1984; Best

1989; Barr 2010), ‘‘who, though able to pass examinations,

have not the necessary aptitude, character or staying power for

a medical career’’ (Goodenough Committee 1944).

For decades medical schools have tried to appraise the

personal qualities that might underpin students’ future

Correspondence: Professor David Powis, School of Psychology (Psychology Building), The University of Newcastle, Callaghan, New South Wales

2308, Australia. Tel: +61 2 4921 5625; E-mail: david.powis@newcastle.edu.au

ISSN 0142-159X print/ISSN 1466-187X online/15/000001–13 � 2015 Informa UK Ltd. 1DOI: 10.3109/0142159X.2015.1009023

Med

Tea

ch D

ownl

oade

d fr

om in

form

ahea

lthca

re.c

om b

y U

nive

rsity

of

New

cast

le o

n 02

/10/

15Fo

r pe

rson

al u

se o

nly.

professional behaviour, by evaluating applicants’ personal

statements and referees’ reports, measuring cognitive skills and

conducting face-to-face interviews, in addition to assessing

their academic suitability. These approaches have not as yet

shown convincing predictive validity for medical school or

later (Gray et al. 2002; Ferguson et al. 2002, 2003; Groves et al.

2007; Lynch et al. 2009; Poole et al. 2012; Kelly et al. 2013). No

previous study has examined the predictive power of non-

cognitive tests, either alone or combined with a range of other

selection tools, throughout the length of a medical course.

However, demonstrating predictive validity of medical student

selection is particularly difficult because most medical schools

record course outcomes only in terms of the results of

academic and clinical examinations. Such outcomes are

seldom in an appropriate form to reflect, or sufficiently

robust to evaluate, students’ non-cognitive and behavioural

attributes (Schuwirth & Cantillon 2005), and seldom include

final summative or barrier assessments of professional

behaviour.

The aim of our study was to examine what student

attributes and qualities, alone or in combination, best predicted

a range of outcomes of medical education. We therefore

undertook an in-depth longitudinal study of an entire entry

cohort of medical students through a five-year medical course

at one UK medical school, in whom most potential predictors

were measured either before or soon after the start of the

course. This longitudinal study explores the question of

predictive validity of the selection methods employed, and

of other potential selection tests, in relation not only to the final

examination outcomes but also to clinical and professional

behaviour throughout the course. The approach was not

hypothesis-driven but exploratory, thus allowing for emergent

relationships between variables.

The data included all the initial selection parameters, plus

results from cognitive and non-cognitive test results not used

in selection, as well as summative written and clinical

examination results from each sitting over the five years of

the course. These examination data allowed novel ways of

exploring clinical performance, because the clinical examin-

ations in the final two years (Objective Structured Long

Examination Reports, OSLERs, and Objective Structured

Clinical Examinations, OSCEs) provided not only measures

of success but also of deficiencies in various aspects of clinical

performance, expressed as ‘‘penalty points’’ (PPs). In addition,

data were collected from regular structured observations made

by students’ tutors (all of whom were experienced clinicians,

the problem-based learning tutors in the first two years as well

as the clinical placement tutors in Years 3–5), which included

assessments of professional behaviours. Uniquely, the tutor

data were collated with the records of the school’s fitness-to-

practise committee to provide a measure of lapses of profes-

sional behaviour, quantified for the purpose of this study on

a scale of ‘‘fitness to practise penalty points’’ (FTPPPs).

We considered that this wide range of assessments, and in

particular the design of the clinical examinations, would

provide a range of measures not only of students’ academic

prowess, but also of their likely personal and behavioural

qualities as clinicians. The exact assessment methods will be

described in detail.

This complex longitudinal data set was analysed to

determine associations between predictor variables (including

demography, prior academic qualifications, cognitive and non-

cognitive/personality and behavioural qualities), and to link

these to criterion variables measured during or at the end of

the course reflecting academic and clinical performance, and

to professional behaviour. The findings from Years 1 and 2 of

the five-year course have been reported earlier (Adam et al.

2012); this report deals with Years 3 to 5 inclusive.

Methods

Study sample

All students admitted to Hull York Medical School (HYMS) in

September 2007 participated in the study, which had ethics

approval from HYMS’ Medical Education Ethics Committee.

HYMS offers a five-year, problem-based, spiral curriculum

within which a large proportion of the clinical experience is

met in primary care. The cohort comprised 146 individuals, of

whom 140 agreed to complete the non-cognitive tests. There

were 62 males (43%) in the study sample. One hundred and

eleven (76%) of the sample were aged under 21 years at the

time of entry and 120 (82%) were British citizens. Ethnic origin

was not recorded. The mean age at entry of the study sample

was 19.9� S. D. 3.9 years (range: 18–42, median age 18).

Data collection

HYMS selection parameters

UCAS form. Selection into the HYMS medical school

programme for this cohort was based on a score derived

from information contained in the Universities and Colleges

Admissions Service (UCAS) form submitted centrally by UK

university applicants, and an interview score.

Practice points

The study uniquely documents� the progress of a complete entry cohort of students

throughout a five-year medical course.� The consequences of an extended range of predictor

variables (including non-cognitive qualities) measured

at the start of the course.� student progress on an extended range of outcome

data collected during the course, including.� frequently repeated behavioural observations

documented by Year 1 and Year 2 tutors.� standardised observation and reporting of all

lapses in professional behaviour.� summative course outcome measures that distin-

guish between performance in clinical assess-

ments and academic examinations.

Together, these features have allowed the creation of a

correlation matrix to determine the strength of linkage

between entry, progress and outcome variables.

J. Adam et al.

2

Med

Tea

ch D

ownl

oade

d fr

om in

form

ahea

lthca

re.c

om b

y U

nive

rsity

of

New

cast

le o

n 02

/10/

15Fo

r pe

rson

al u

se o

nly.

HYMS academic score. Each applicant’s UCAS form

listed their academic results (either predicted or already

obtained), usually as either advanced-level grades (or equiva-

lent) or a degree classification. The academic grades likely to

be required for entry were published before applications

opened. Administrative staff categorised the academic results

listed and determined a numerical HYMS academic score,

using guidelines that ensured comparability with other systems

of examination results such as the International Baccalaureate.

The HYMS academic scores, using the example of A level

grades, were: Unsatisfactory¼ 0, predicted 2 A level grades

below likely requirement; Just satisfactory¼ 2, predicted 1 A

level grade below likely requirement; Clearly satisfactory¼ 5,

predicted or obtained likely requirement; Outstanding¼ 8,

above likely requirement, either already obtained or predicted

(in which case supporting evidence was required from earlier

examination results). Only applications with a HYMS academic

score 40 were considered further. The mean academic score

in the study cohort was 6.1 (range 2–8, median 5 and mode 8).

Personal statement and referee’s report. The remain-

ing UCAS applications were assessed by two independent

readers using a structured information sheet to assess the

referees’ comments and record evidence of a realistic under-

standing of medicine, examples of self-motivation, acceptance

of responsibility, communication skills and team working, and

any other distinctive attributes (e.g. social disadvantage or

exceptional sporting achievement). The data were then scored

by administrative staff. The average score per reader was 25

(range 17–35).

The HYMS academic score and mean of the two reader

scores were added together to give the UCAS form score,

maximum 50 points.

Interview scores. The applicants with the top UCAS form

scores were invited to interview. Structured, scripted 20-

minute interviews were conducted by two interviewers,

yielding a maximum possible interview score of 50, compris-

ing 40 points based on the answers to eight questions and 10

points from an overall assessment of suitability scored on a 0 to

10-point visual analogue scale.

The UCAS form score and interview score were added to

contribute equally to the final selection score on which

applicants were ranked; places were then offered to the top

scoring candidates.

Potential selection instruments

Several standardised instruments were administered to this

cohort as part of this study, but were not used in selection.

Traits and skills measured by the cognitive UK Clinical

Aptitude Test (UKCAT) and non-cognitive tests (Personal

Qualities Assessment, PQA; Resilience Scales Questionnaire,

RSQ; Trait Emotional Intelligence Questionnaire, TEI) are

written in italics. Combined traits derived from these are

written IN ITALIC CAPITALS.

Cognitive tests

UKCAT. The UKCAT is a mandatory standardised test of

cognitive ability for those applying to study medicine at the

majority of UK medical schools. It was first taken in 2006, by

applicants to courses starting in 2007 (see www.ukcat.ac.uk).

UKCAT results were available for 131 students; 13 had applied

the year before the test was introduced, one student was

exempt and one result was unavailable. The UKCAT

comprised four cognitive skills subtest scores: verbal reason-

ing (VR), numerical reasoning (NR), abstract reasoning (AR)

and decision analysis (DA) (Childs 2012) and a total score. The

UKCAT scores were not revealed to the HYMS selectors at any

stage and did not inform selection decisions.

Non-cognitive tests

PQA. This comprised three tests:

(a) The Interpersonal Traits Questionnaire, which measures

the traits narcissism, aloofness, confidence (in dealing

with people) and empathy and produces a summary

score for the combined trait INVOLVEMENT (versus

DETACHMENT) in which confidence and empathy are

positive, narcissism and aloofness negative (Munro et al.

2005).

(b) The Interpersonal Values Questionnaire, which measures

the extent to which the respondent favours individual

freedoms (versus societal rules) as the basis of their moral

orientation (Bore et al. 2005).

(c) The Self-Appraisal Inventory, which measures the com-

bined traits EMOTIONAL RESILIENCE (comprising scales

measuring anxiety, moodiness, neuroticism and irra-

tional thinking) and SELF-CONTROL, in contrast to risk

taking tendency, (using the scales of restraint, conscien-

tiousness, permissiveness and anti-social tendencies). The

inventory also contains a Lie scale (Bore et al. 2009;

www.pqa.net.au).

RSQ. RSQ is a self-report questionnaire that identifies six

cognitive, behavioural and affective components, named

self-esteem, optimism, self-discipline, control, emotional non-

defensiveness and image management (impression manage-

ment) (Childs 2012).

TEI. The TEI Short Form is a 30-item questionnaire designed

to measure the global trait EMOTIONAL INTELLIGENCE

(Petrides 2009; Cooper & Petrides 2010).

The non-cognitive tests were all delivered in a paper-based

format under examination conditions at the University of Hull

and the University of York; PQA and TEI in October 2007 and

RSQ in October 2008.

Tutor ratings and grades

Problem-based learning tutors in Years 1 and 2 met with their

students twice weekly for 90 minutes. They assessed the

individual students’ interpersonal skills and professional

behaviours, which were recorded on standard scales in Year

1 (May 08) and Year 2 (January 09 and May 09), as described

in Adam et al. (2012). ‘‘Tutor ratings’’ represent the sum of the

scores for each of 14–17 defined skills or behaviours. At the

same time, these experienced problem-based learning tutors

made a global assessment, grading each student as either

Predictors of outcome at medical school

3

Med

Tea

ch D

ownl

oade

d fr

om in

form

ahea

lthca

re.c

om b

y U

nive

rsity

of

New

cast

le o

n 02

/10/

15Fo

r pe

rson

al u

se o

nly.

‘‘problematic’’, ‘‘average’’ or ‘‘particularly promising’’. This is

the ‘‘tutor grade’’.

Course summative assessments

Course assessments were undertaken in accordance with

HYMS’ Code of Practice on Assessment and Examination for

MB BS in Phases II & III. There was no summative assessment

in Year 3. The Year 4 and Year 5 summative assessments are

summarised below.

Year 4 written examination. Marks in the written examin-

ations at end of academic year 4 were allocated to one of two

HYMS themes: Theme A (Life sciences and Clinical sciences)

and Theme C (Evidence-based decision-making, Population

health and medicine and Managing resources for quality and

efficiency). The papers were mapped to course outcomes

including those from pharmacology and therapeutics. Theme

C tested not only application of clinical knowledge, but also

analytic and numerical evaluation skills across a range of

medical contexts. Theme B (Clinical Techniques and Skills;

Person-Centred Care) was examined in Year 4 only by the

clinical examinations.

Year 5 written examination. The written examination at

the end of academic year 5 was an integrated paper covering

all aspects of the course (Themes A, B and C). The questions

were based on clinical scenarios, each one orientated around a

common management problem including preventive strate-

gies. Therapeutic issues were a major focus, but other

management issues were also examined in this paper. The

pass mark was determined by the Hofstee method (McKinley

& Norcini 2014). Students sat the paper in March and those

who did not achieve the pass mark took another paper in May.

This was an independent paper but it was not considered a

resit. If the students passed at the second attempt, then they

passed the written examination.

Year 4 and Year 5 clinical examinations. The Year 4 and

Year 5 clinical examinations used a number of Objective

Structured Long Examination Records (OSLERs), each being a

45-minute observed clinical assessment in which the student

met and talked to a real patient, undertook appropriate

examinations, spent 15 minutes alone preparing a written

summary and plan, and then discussed this with the two

examiners and the patient. Student performance was assessed

in four categories of competence in Year 4: gathering

information, clinical examination, problem solving, and

relationship with patient. A fifth category, patient manage-

ment, was added in Year 5.

The Year 5 examinations also included Objective Structured

Clinical Examinations (OSCEs), which were 7-minute stations

assessed by direct observation by one examiner, the majority

designed to test students’ high level ‘‘communication skills’’

using simulated patients.

Practical clinical procedures were not routinely included

in the clinical examinations. Each student was required to

have reached a satisfactory standard in every specified

practical clinical procedure, assessed earlier in a controlled

clinical environment, before being allowed to take the

OSCEs.

The HYMS clinical examinations used a sequential design

and a non-compensatory marking system, fully described

elsewhere (Cookson et al. 2011). The key features are

summarised below. The sequential design required all students

to take the first part of the examination. The best performing

candidates (usually around 70%) were found to be clearly

satisfactory at this point. The remaining approximately 30%

were required to take the second part of the examination; the

additional results from the second part, together with the

results from the first part, were summed to give increased

reliability in determining which side of the pass/fail boundary

each student lay. Around 5–8% eventually failed the examin-

ation. The Year 4 clinical examinations consisted of two

OSLERs in the first part and three OSLERs in the second part.

The Year 5 examinations comprised four OSLERs and six

OSCEs in the first part, and the same number in the second

part. Four of the six OSCEs in the first part and five of the six

OSCEs in the second part addressed communications skills; the

remainder involved a written task or skill.

Grading and marking system

Each category of competence examined in the OSLERs, and

every OSCE, was graded by the examiners using the following

grade descriptors shown in Table 1.

The outcome of these examinations was pass/fail only. On

the basis that candidates should not be able to compensate for

Table 1. Grade descriptors, PPs and scores for OSLERs and OSCEs.

Grades DescriptorPenalty pointsYear 4

Penalty pointsYear 5

Researchscore

A Capable in all components to a high standard – – 6

B Capable in all components to a satisfactory standard and

a high standard in many

– – 5

Cþ Capable in all components to a satisfactory standard – – 4

C� Capable in a majority of components to a satisfactory

standard, inadequacies in some components

– 1 3

D Capable in a minority of components.

No serious defects

2 2 2

E Capable in a minority of components.

One or more serious defects

3 3 1

J. Adam et al.

4

Med

Tea

ch D

ownl

oade

d fr

om in

form

ahea

lthca

re.c

om b

y U

nive

rsity

of

New

cast

le o

n 02

/10/

15Fo

r pe

rson

al u

se o

nly.

a serious deficiency in one area by a high standard in another,

grades below satisfactory were converted to PPs, and only the

PPs were used to make the pass/fail decisions. Candidates

accruing more than a fixed number of PPs in the first part were

required to take the second part of the examination. HYMS

determined the examination outcome from the total number of

PPs accrued, across either the first part or both parts of the

examination if taken, to give a pass/fail result. The grades A to

E form an ordinal categorical scale (Cookson et al. 2011), so for

the purpose of this study they were converted to positive

scores, as shown in Table 1. For this study we have expressed

all examination data as an average per case, derived from the

scores from the first part of the examination, and from the

second part if taken, whether positive scores or PPs. These are

designated OSLER score, OSCE score, OSLER PPs and OSCE

PPs. A further summary score and penalty point measure was

calculated for the entire Year 5 clinical examination by adding

together the OSLER and OSCE scores (equally weighted) and

the OSLER and OSCE PPs (equally weighted), called OSLER þOSCE sum score and OSLER þ OSCE sum PPs, respectively.

The maximum number of possible PPs (designated Clinical

Examination penalty points, CEPPs) from all the Year 5

OSLERs and OSCEs adding the first part and the second part

together was 276; the observed range in this cohort was 0 to

56, mean 11.04, median 6, mode 4.

Other outcome measures

Other relevant outcome measures detailed here include

subscales derived from the summative examinations that

were considered particularly relevant to explore for this

study, such as communication skills and Honours awards,

and empathy scores provided as formative examination

feedback to the students.

OSLER communication scores. The OSLER competence

categories ‘‘gathering information’’ and ‘‘relationship with

patient’’ rely heavily on communication skills. The scores

and PPs accrued for these competences alone were added

together and a mean score per case calculated as described

above, to give the OSLER communication score and OSLER

communication PPs.

OSCE communication scores. Four of six OSCE stations in

the part 1 examination and 5 of 6 in the part 2 examination

assessed communication skills in challenging situations; the

mean score and PPs per case accrued from only these stations

yielded the OSCE communication score and OSCE communi-

cation PPs.

OSCE empathy scores. At each communication OSCE

station, the examiner and simulated patient undertook inde-

pendent assessments of the student’s perceived empathy,

graded A to E, ranging from ‘‘excellent’’ to ‘‘poor’’ empathy

skills (Wright et al. 2014). These grades were collected for

research, and provided to students as formative feedback. For

the present study the grades were converted to empathy

scores (A, B, C, D, E¼ 5, 4, 3, 2, 1 points) and PPs (D, E¼ 1, 2

PPs) and used to calculate a mean OSCE Empathy score and

OSCE Empathy PPs per case, using data from both parts of the

examination.

The clinical examination variables thus fall into two

categories, those that give positive scores for performance,

and those that give negative scores for deficient performance.

Better performance is denoted by higher totals in the OSLER

score, OSLER communication score, OSCE score, OSCE

communication score, OSCE Empathy score and OSLER þOSCE sum score. The indices of deficient clinical performance

are OSLER PPs, OSLER communication PPs, OSCE PPs, OSCE

communication PPs, OSCE Empathy PPs, OSLER þ OSCE sum

PPs and clinical examination PPs.

Criteria for graduation with honours

HYMS applied specific criteria for recommending students for

graduation with Honours, based on weighted overall perform-

ance in summative examinations and student projects through-

out the course. For this cohort approximately 6% were

awarded Honours.

Fitness to practise penalty points

The final important outcome measure was FTPPPs. The HYMS

Fitness-to-Practise committee received confidential reports (in

which students were identified only by number) not only

about serious lapses in professional behaviour, but also about

lower level concerns arising from structured formative end-of-

block reviews. These reviews were undertaken regularly

between student and their current clinical tutor at approxi-

mately two monthly intervals throughout the entire course,

and included grading of the student’s professional behaviour

(using a structured score sheet with clear grade descriptors),

either Excellent, Satisfactory, Borderline or Unsatisfactory,

under each of the following headings: ‘‘relationships with

patients’’; ‘‘awareness of ethical and moral aspects of sub-

ject’’; ‘‘ability to deal with uncertainty and awareness of

limitations’’; ‘‘evidence of self-education, enthusiasm and

motivation’’; ‘‘teamwork’; ‘‘dress, attendance and punctuality’’.

A ‘‘borderline’’ or ‘‘unsatisfactory’’ grade in any aspect of

professional behaviour was automatically notified to the

Fitness-to-Practise Committee. This allowed HYMS to identify

and react when students showed repeated patterns of

undesirable behaviour.

J. A. reviewed all the reports of the HYMS Fitness-to-

Practise committee covering the HYMS academic years 2007–

2013 inclusive, and summarised every instance concerning a

member of the study cohort. Each instance was given one or

more FTPPPs devised for this study; the total number of points

accrued by each student across the years of the course was

then recorded. For example, a professional behaviour grade of

‘‘borderline’’ was given 1 FTPPP and an ‘‘unsatisfactory’’ grade

was given 2 FTPPPs. Other misdemeanours were awarded one

or two points, after assessing their seriousness through careful

evaluation of all the relevant information including contem-

poraneous verbal accounts and written records. In total, 45

individuals were identified in Fitness-to-Practise records, of

whom 26 had only 1 FTPPP. There were long-term conse-

quences for a significant proportion of those accruing three or

Predictors of outcome at medical school

5

Med

Tea

ch D

ownl

oade

d fr

om in

form

ahea

lthca

re.c

om b

y U

nive

rsity

of

New

cast

le o

n 02

/10/

15Fo

r pe

rson

al u

se o

nly.

more FTPPPs, with referral to formal fitness-to-practise pro-

cedures that can impose serious penalties, the most serious

being either a formal warning reported to the UK General

Medical Council on registration as a doctor, or being required

to leave the course.

Data handling and statistical analysis

All data were anonymised and entered into an SPSS database.

All analyses utilised SPSS version 20 (Chicago, IL). Scores on all

multiple-component measures (OSCEs, OSLERs, communica-

tion and empathy measures, etc.) were computed and checked

for internal consistency. Except where noted (e.g. age) no

extreme skewness and no extreme outliers were detected.

Relationships between continuous variables were computed as

Pearson correlation coefficients (r); one-way analysis of

variance (ANOVA, F-statistic) was used to compare scores on

categorical variables (sex, citizenship, etc.) and on compari-

sons of extreme groups, such as top versus bottom 20% of

scores. Statistical significance was based on unadjusted

probabilities (p¼ 0.05, 0.01 or 0.001). Two-tailed tests of

significance were applied in most cases, except where a clear

directional relationship was predicted. In order to establish the

best overall predictors of outcomes, a series of linear regres-

sion analyses were undertaken and are described in the

Results section.

Results

The tables show all the statistically significant results. Table 2

shows the significant correlations found between the initial

predictor variables (available from the start of the course),

intermediate tutor evaluations (from Years 1 and 2), demo-

graphic factors and the outcomes from the Year 4 and Year 5

written and clinical examination results. We considered that

graduating with Honours or with notably high written and

clinical examination results were both desirable selection

outcomes, whereas failing to complete the course or having

documented instances of significant unprofessional behaviour

were undesirable selection outcomes. Table 3 shows which

predictors were significant when ANOVA was applied to three

pairs of comparisons based on positive exam marks: those

students who graduated with Honours (n¼ 11) versus those

who left the course (n¼ 9); those students with Year 5 written

exam scores in the top quintile versus the bottom quintile;

Table 2. Predictors of examination scores.

Initial predictors Intermediate predictors Demographic predictors

HYMS academic UKCAT PQA RSQ HYMS tutors

Years 1 & 2

examinations Age Sex UK/Non-UK

Written exams

Year 4 theme A r 0.174* Jan 09g r 0.261**

May 09g r 0.257**

08þ09g r 0.236**

f (F)**

Year 4 theme C r 0.200* Tot r 0.181* Jan 09g r 0.246**

May 09g r 0.218*

08þ09r r 0.194*

08þ09g r 0.239**

y (F)* UK (F)*

Year 5 final exam Tot r 0.175*

AR r 0.231**

f (F)* UK (F)*

Clinical exams

Year 4 OSLER score ASoc r �0.181* 08þ09r r 0.179*

Year 4 OSLER communication

score

o (F)* f (F)*

Year 5 OSLER score r 0.213* Lie r 0.198*

Emp r 0.257**

En-d r 0.219* Jan 09g r 0.176*

May 09r r 0.176*

May 09g r 0.197*

08þ09g r 0.197*

y (F)** f (F)**

Year 5 OSLER Communication

score

r 0.215* Conf r 0.192*

Emp r 0.260**

(TEI r 0.201)* f (F)*** UK (F)*

Year 5 OSCE score r 0.170* Tot r 0.204*

VR r 0.244**

AR r 0.250**

y (F)** UK (F)**

Year 5 OSCE Communication

score

Jan 09r r 0.185*

May 09r r 0.188*

Year 5 empathy score UK (F)*

Year 5 OSLERþOSCE

sum score

r 0.256** Emp r 0.194* May 09r r 0.191*

May 09g r 0.190*

Year 1A r 0.254**

Year 1B r 0.228**

Year 1C r 0.215*

Year 1T r 0.296***

Year 2A r 0.301***

Year 2B r 0.337***

Year 2C r 0.271**

Year 2T r 0.359***

y (F)* f (F)* UK (F)**

Analysis of variance (F statistic) for categorical variables and Pearson correlations (r) for continuous variables, *p50.05; ** p50.01; *** p50.001 (N¼131–146;

significance levels are not adjusted for repeated comparisons).

UKCAT Tot, total UKCAT score; VR, verbal reasoning; AR, abstract reasoning; PQA Antisoc, anti-social tendencies; Lie, lie scale score; Emp, empathy; Conf,

confidence; RSQ En-d, emotional non-defensiveness; TEI, test of emotional intelligence. HYMS tutors: ratings, r, and grades, g, given in years 1 (May 08) and 2

(January 09, May 09). Years 1 & 2 exams: themes A, B and C, and totals, T, of AþBþC, y: younger; o: older at entry; f: female; UK: UK citizen.

J. Adam et al.

6

Med

Tea

ch D

ownl

oade

d fr

om in

form

ahea

lthca

re.c

om b

y U

nive

rsity

of

New

cast

le o

n 02

/10/

15Fo

r pe

rson

al u

se o

nly.

those students with Year 5 clinical examinations (OSLER þOSCE sum) scores in the top quintile versus the bottom

quintile. Table 4 shows the significant predictors of deficient

clinical performance or undesirable outcomes, from correl-

ations with clinical examination PPs, and from comparisons of

three groups of students achieving undesirable outcomes:

those who left the course versus the rest; those who left the

course grouped with those who gained 3 or more FTPPPs

versus the rest, and those who gained any FTPPPs versus those

who did not. Finally, Table 5 shows the outcome of the

multiple regression analyses to determine the best predictors

of Year 4 and Year 5 outcomes.

Predictors of achievement and progress

Demographic data

Age. Students under 21 at the time of entry performed

better than older students in the Theme C component of the

Year 4 written examination and in the Year 5 clinical

examinations (Year 5 OSLER, Year 5 OSCE). Older students

outperformed their younger peers only in the Year 4 OSLER

examination, specifically in the ‘‘communication’’ segments

(Table 2).

Sex. Females performed better than males in the Theme A

component of the Year 4 written examination and in the

Year 5 final written examination (an integrated examination

covering Themes A, B and C). They outperformed males also

in Year 4 and Year 5 OSLERs, and achieved a better overall

score in the combined Year 5 OSLERþOSCE sum score

(Table 2). They received fewer OSLER and OSCE PPs than

their male peers (Table 4). Females were better represented

than males in the top 20% of achievers in the final year, in both

written and clinical examinations (Table 3). Males did not

outperform females in any component of the Year 4 and 5

examinations.

Citizenship. Non-UK citizens performed at a lower level in

Year 4 and 5 written examinations and the Year 5 clinical

examinations than UK citizens (Table 2), and gained signifi-

cantly more PPs and penalty marks in all of the clinical

examinations (Table 4).

HYMS selection process

The HYMS academic score, an index of prior academic

performance, was the only item used in the selection

procedure that was a useful, significant predictor of progress,

correlating with a number of assessment outcomes. Written

exam performance (Year 4, Themes A and C) and clinical

exam performance were predicted (Year 5 OSLER and Year 5

OSCE) (Table 2). The HYMS academic score also correlated

(negatively) with the Year 5 OSLERþOSCE sum PPs and the

total clinical exam penalty points (CEPPs) (Table 4). No other

Table 3. Predictors of top and bottom performers.

Initial predictors Intermediate predictors Demographic predictors

HYMSacad UKCAT PQA RSQ HYMS tutors Years 1&2 exams Age Sex UK or non-UK

Graduation

with Honours

versus left course

May 08r (F)*

Jan 09g (F)**

May 09g (F)***

Year 5 written examination Tot (F)* Yr 1A (F)*** f (F)*

Top 20% versus bottom 20% VR (F)**

AR (F)**

QR (F)**

Yr 1C (F)***

Yr 1 T (F)***

Yr 2 OSCE (F)*

Yr 2A (F)***

Yr 2B (F)***

Yr 2C (F)***

Yr 2 T (F)***

Year 5 clinical examination

(OSLERþOSCE sum score)

Top 20% versus bottom 20%

(F)* Mood (F)*

Consc (F)*

Conf (F)**

(TEI (F)*)

En-d (F)*

May 09r (F)*

May 09g (F)*

08þ09r (F)*

08þ09g (F)*

Yr 1 OSCE com (F)**

Yr 1 OSCE prac (F)*

Yr 1 OSCE T (F)**

Yr 1A (F)**

Yr 1B (F)**

Yr 1C (F)*

Year 1 T (F)***

Yr 2 OSCE (F)***

Yr 2A (F)***

Yr 2B (F)***

Yr 2C (F)**

Year 2 T (F)***

y (F)* f (F)**

Analysis of variance (F statistic) for categorical variables, *p50.05; **p50.01; ***p50.001 (N¼131–146; significance levels are not adjusted for repeated

comparisons).

HYMS acad: computed academic entry score, see Methods section; UKAT Tot, total UKCAT score; VR, verbal reasoning; AR, abstract reasoning; QR, quantitative

reasoning; PQA Mood, moodiness; Consc, conscientiousness; Conf, confidence. RSQ En-d, emotional non-defensiveness. TEI, test of emotional intelligence.

HYMS tutors: ratings, r, and grades, g, given in years 1 (May 08) and 2 (January 09, May 09). Year 1 & 2 exams: themes A, B and C, and total, T, of AþBþC, and

OSCE practical and communication skills stations and sum total, T, of ‘‘prac’’ and ‘‘com’’. y: younger age at entry; f: female.

Predictors of outcome at medical school

7

Med

Tea

ch D

ownl

oade

d fr

om in

form

ahea

lthca

re.c

om b

y U

nive

rsity

of

New

cast

le o

n 02

/10/

15Fo

r pe

rson

al u

se o

nly.

variables derived from the UCAS form personal statement &

referees’ report score, nor variables from the HYMS interview

question scores and visual analogue scale, were useful

predictors; indeed, ‘‘interview score’’ correlated significantly

only with Year 4 OSLER communication PPs, but in the

unexpected direction (Table 4).

Cognitive tests: UKCAT sub-scores and total score

UKCAT total score predicted some written exam scores (Year

4, Theme C and Year 5) and clinical performance assessed by

Year 5 OSCE. Some UKCAT subtest scores (Abstract

Reasoning and Verbal Reasoning) were also significantly

correlated with the written and some clinical exam scores in

Year 5, but not Year 4 (Table 2).

Non-cognitive tests: PQA

Many significant correlations were found between some trait

and combined trait scores and clinical exam scores, CEPPs and

FTPPPs. In contrast, none of the PQA test scores predicted

written exam performance in Years 4 and 5 (Table 2). The

PQA traits confidence, conscientiousness, anti-social tenden-

cies, empathy, moodiness, neuroticism and aloofness and the

combined trait INVOLVEMENT were shown to be predictors of

clinical examination scores (Tables 2 and 3) and of PPs in

these exams (Table 4).

Non-cognitive Tests: RSQ

Like PQA, some RSQ components (emotional non-defensive-

ness, self-esteem and optimism) predicted clinical performance

both good and bad, but not written exam performance

(Table 2).

Intermediate predictors: In-course Tutor ratings and

grades

Tutor evaluations (both ratings and grades) of students’

capabilities and deficiencies made during three terms in

Table 4. Predictors of undesirable outcomes.

Initial predictors Intermediate predictors Demographic predictors

HYMS

acad

HYMS

interview UKCAT PQA RSQ HYMS tutors

Years 1&2

exams Age Sex UK or non-UK

Year 4 OSLER PPs 08–09g r �0.192*

May 08g r �0.185*

Year 4 OSLER

Communication PPs

r 0.190* Neurot r �0.177* E n-d r 0.211*

Year 5 OSLER PPs r �0.187* Emp r �0.255**

Inv r �0.187*

m (F)* Non (F)*

Year 5 OSLER

Communication PPs

Emp r �0.241** m (F)***

Year 5 OSCE PPs VR r �0.218**

AR r �0.274**

QR r �0.261**

Tot r �0.304***

Conf r �0.198* Self-est r �0.220*

Optm r �0.231*

Non (F)*

Year 5 OSCE

Communication PPs

VR r �0.177*

AR r �0.196*

Conf r �0.195* May 08g r �0.175*

08–09g r �0.187*

Non (F)*

Year 5 OSCE

Empathy PPs

Non (F)*

Year 5 CEPPs r �0.235** Emp r �0.369***

Inv r �0.297**

08–09r r �0.177* m (F)* Non (F)*

Year 5

OSLERþOSCE

sum PPs

r �0.208* VR r �0.214* Emp r �0.179* Yr 1A r �0.251**

Yr 1B r �0.180*

Yr 1C r �0.230**

Yr 1T r �0.284***

Yr 2A r �0.288***

Yr 2B r �0.338***

Yr 2C r �0.275***

Yr 2T r �0.352***

m (F)** Non (F)**

Left course

(versus or not)

Antisoc (F)* Optm (F)* May 08 (F)*

Jan 09r (F)*

Jan 09g (F)**

May 09r (F)**

May 09g (F)**

08–09r (F)*

Yr 2C (F)***

Yr 2T (F)*

Yr 2 OSCE (F)**

o (F)**

Left course or43

FTPPPs versus

‘‘not’’ and ‘‘none’’

Antisoc (F)* Jan 09r (F)**

08–09r (F)*

Yr 2 OSCE (F)*

Yr 2C (F)**

Yr 2 T (F)*

o (F)*

FTPPPs (versus none) Aloof (F)* Image (F)* Yr 1B (F)*

Yr 1 OSCE (F)*

Yr 2B (F)*

Yr 2C (F)**

m (F)*

Analysis of variance (F statistic) for categorical variables and Pearson correlations (r) for continuous variables, *p50.05; **p50.01; ***p50.001 (N¼ 131–146;

significance levels are not adjusted for repeated comparisons).

UKCAT Tot, total UKCAT total score; VR, verbal reasoning; AR, abstract reasoning; QR, quantitative reasoning; PQA Antisoc, anti-social tendencies; Aloof, aloofness;

Emp, empathy; Conf, confidence; Neurot, neurotic; Inv, INVOLVED. RSQ En-d, emotional non-defensiveness; Optm, optimism; Image, managing own image.

Self-est, self-esteem. HYMS tutors: ratings, r, and grades, g, given in years 1 (May 08) and 2 (January 09, May 09). Year 1&2 exams: themes A, B and C, and totals,

T of AþBþC, and OSCE; o: older at entry; m: male; f: female; UK/Non: UK citizen or non-UK citizen. PP: penalty points; FTPPP: fitness to practise penalty points.

J. Adam et al.

8

Med

Tea

ch D

ownl

oade

d fr

om in

form

ahea

lthca

re.c

om b

y U

nive

rsity

of

New

cast

le o

n 02

/10/

15Fo

r pe

rson

al u

se o

nly.

Years 1 and 2 were not only significant predictors of many

components of written and clinical examination performance

in Years 4 and 5 (Table 2), but also significant predictors of the

groups of students who gained Honours or left the course

(Tables 3 and 4).

Intermediate predictors: Year 1 and Year 2 examina-

tion performance

Year 1 Themes A, B and C results and Year 2 Themes A, B and

C results each individually predicted Year 5 OSLERþOSCE

sum score (Table 2) and Year 5 OSLERþOSCE sum PPs (Table

4). Themes B and C were also found to significantly predict

who left the course and FTPPPs (Table 4). The top and bottom

20% of achievers in Year 5 written and clinical examinations

(OSLER & OSCE) were generally predicted by Year 1 and 2

results (Table 3). However, Year 1 and 2 examination results

did not predict graduation with Honours or leaving the course

(Table 3).

Desirable and undesirable outcomes

Graduation with honours

By 2013, nine individuals (6%) from the study sample had

graduated with honours, seven of whom were aged 521 at

entry, three were males and eight were UK citizens. Year 1

and 2 tutor ratings (r) and grades (g) were the only significant

independent predictors of ‘‘graduation with honours’’

(Table 3).

Leaving the course

Eleven individuals from the study sample (8%) left the course

without completing their medical degree; six (55%) were aged

521 at entry, two (18%) were males and eight (72%) were UK

citizens. Those who left scored more highly on RSQ optimism

and PQA anti-social tendencies. Year 1 and 2 tutor ratings and

grades (May 08, January 09 and May 09) were consistent

predictors of those who left the course (Table 4).

Comparison of those who gained Honours with those

who left the course

The only significant predictors of leaving the course versus

graduating with honours were Year 1 (May 08) tutor ratings

and Year 2 (January 09, May 09) tutor grades (Table 3).

Comparison of top 20% with the bottom 20% of

achievers in the Year 5 final examinations

Year 5 written examination. Being among the top 20% of

achievers was significantly associated with being female, or

having a higher UKCAT total score (and better UKCAT

quantitative reasoning, verbal reasoning and abstract rea-

soning scores). Students who had performed to a higher

standard in Year 1 and Year 2 examinations were significantly

more likely to be represented in the top scoring group in the

Year 5 written examination (Table 3).

Year 5 clinical examination. The following variables

significantly indicated the likelihood of being among the top

20% of achievers: being female, younger and scoring better in

some of the non-cognitive tests (PQA: lower moodiness, higher

conscientiousness and higher confidence; RSQ: higher emo-

tional non-defensiveness; TEI: higher emotional intelligence).

Better tutor ratings and grades in Year 1 and 2 predicted higher

achievement, as did better overall marks and individual

components by theme of Year 1 and Year 2 examinations

(Table 3).

Comparison of those with extreme high or low non-

cognitive PQA scores with central scores

Bore et al. (2009) proposed that extreme scores (both high and

low) on the PQA non-cognitive tests would be likely to predict

low scores on positive course outcomes and/or high scores on

negative course outcomes. In order to test this hypothesis,

outcome data from the students in the top and bottom 20% of

scores on each non-cognitive PQA trait were combined and

compared with the middle 60%, using analysis of variance. The

hypothesis was confirmed for the PQA combined trait

INVOLVEMENT in relation to the Year 5 OSCE score

(p¼ 0.036), Year 5 OSCE empathy score (p¼ 0.042) and also

Year 5 OSCE empathy PPs (p¼ 0.035).

Comparison of those reported to the fitness-to-practise

committee with the rest of the cohort

None of the HYMS selection criteria predicted incidents or

behaviours of concern to the Fitness-to-Practise committee.

Males were significantly more likely to gain FTPPPs, but

neither age at entry nor UK citizenship were significant

predictors. Those students who had underperformed in

components of the Year 1 and Year 2 examinations were

more likely to be those who gained FTPPPs. Neither UKCAT

sub-scores nor total scores predicted FTPPPs, but PQA

aloofness and RSQ managing image did predict FTPPPs

(Table 4).

Individual characteristics of the subgroup of students

reported to the fitness-to-practise committee

Overall, 45 students gained FTPPPs. ANOVA (F-test) showed

that several predictors characterised this subgroup: UKCAT

higher verbal reasoning scores (p¼ 0.016); PQA lower con-

scientiousness (p¼ 0.023); PQA higher impulsiveness

(p¼ 0.041); PQA higher confidence (p¼ 0.015); RSQ lower

self-discipline (p¼ 0.015); RSQ lower control (p¼ 0.011).

Lower Year 1 and 2 tutor ratings were also a significant

predictor (sum of May 08þ January 09þMay 09 ratings;

p¼ 0.018). A stepwise regression which included all of the

above predictors found that RSQ control was the predominant

predictor (beta¼�0.539; p¼ 0.004). The subgroup with

FTPPPs was also associated with poorer clinical performance:

in Year 4 a lower OSLER score, p¼ 0.004; lower OSLER

communication score, p¼ 0.004; more OSLER PPs and OSLER

communication PPs, p¼ 0.000 and p¼ 0.003, respectively, and

in Year 5 more OSCE PPs (p¼ 0.010) and clinical examination

PPs (p¼ 0.018).

A further comparison of interest from the selectors’

perspective is reported in Table 4. The worst selection

outcomes are students who either leave the course or manifest

Predictors of outcome at medical school

9

Med

Tea

ch D

ownl

oade

d fr

om in

form

ahea

lthca

re.c

om b

y U

nive

rsity

of

New

cast

le o

n 02

/10/

15Fo

r pe

rson

al u

se o

nly.

serious professional misbehaviour. We therefore grouped

together those students who had either left the course

prematurely or who had gained three or more FTPPPs

(n¼ 22), and compared them with the rest of the cohort, to

look for significant predictors of these undesirable outcomes.

This group were older at entry, had scored higher on the PQA

anti-social tendencies measure, and all had scored lower on

the summed Year 1 and 2 tutor ratings (May 08þJanuary

09þMay 09). These students had also performed less well in

components of the Year 2 examination (Table 4).

Best predictors of Year 4 and Year 5outcomes: Regression analyses

To clarify the complex relationships between predictors and

outcomes, a series of linear regression analyses were under-

taken using the SPSS routines, which revealed that the best

overall predictors of all outcomes were the demographic

variables (age, sex and citizenship), the prior ‘‘ability’’ indica-

tors (HYMS academic score and total UKCAT score) and the

non-cognitive test scores for empathy (PQA) and emotional

non-defensiveness (RSQ). The apparent selection power of the

demographic variables is likely to be mediated by other

confounding variables, so the two ability variables (the most

readily available for selection purposes) were entered into the

first stage of a multiple regression analysis, and the two non-

cognitive variables (which might be added to the selection

procedures) in the second stage. The results are shown in

Table 5, which also shows the percentage of variance in each

key outcome variable accounted for by the ability and non-

cognitive predictors. In a third stage, the three demographic

variables were then entered, to show how much additional

variance might be accounted for by the other factors they

represent.

The results show that the prior ability variables alone would

be useful predictors of Year 4 examinations, Year 5 OSCE and

OSLER scores and most of the associated penalty point scores,

but not the Year 4 OSLERs or Year 5 OSCE empathy score. The

HYMS academic score appears to be a better predictor than

the UKCAT total, which in the case of Year 5 OSLER scores

showed a negative relationship with the outcomes. The

addition of the non-cognitive variables would make a signifi-

cant contribution in the case of the Year 5 OSLER scores and

PPs, with PQA empathy and RSQ emotional non-defensiveness

contributing about equally. The difference between the value

of the adjusted R-squared (percentage of the variance) at stage

1 and 2 and after stage 3 shows that the additional

demographic variables, particularly sex and citizenship,

would attenuate the predictive usefulness of the prior ability

and non-cognitive variables.

Discussion

The aim of our study was to examine what student attributes

and qualities, or combination thereof, best predicted outcomes

of medical education. This was undertaken longitudinally in

the context of a specific medical school cohort with predictors

being measured some four to five years prior to measurement

of the outcome variables. The approach was not hypotheses-

driven but exploratory thus allowing the data to reveal

relationships between variables. The principal findings were:

Table 5. Best selectors for Year 4 & Year 5 examination outcomes.

1st stage 2nd stage 3rd stage

Outcome variableUCASacad

UKCATtotal

PQAempathy

RSQe n-d

Stages 1þ2adjusted

R-squared(% of variance) Sex Age UK/Non

Stages 1þ2þ3adjusted

R-squared(% of variance)

Years 4 & 5 – positive outcomes

Year 4 Theme A exam 0.22* 0.08 0.05 0.12 4.3 0.30** 0.00 0.14 10.9

Year 4 Theme C exam 0.31** 0.04 0.06 0.14 8.8 0.14 0.14 0.15 10.5

Year 4 OSLER total �0.02 �0.07 0.13 �0.12 0.0 0.22* 0.19 0.15 2.9

Year 4 OSLER communication �0.01 �0.11 0.01 �0.16 0.0 0.29** 0.22* 0.11 8.0

Year 5 OSCE total 0.14 0.18 0.05 0.02 2.3 �0.04 �0.22* 0.24* 11.6

Year 5 OSCE communication 0.21* 0.05 0.01 0.18 4.2 0.01 0.08 0.17 4.1

Year 5 OSLER total 0.30** �0.22* 0.21* 0.25** 20.1 0.19* 0.12 0.16 23.3

Year 5 OSLER communication 0.24* �0.29** 0.21* 0.20* 17.7 0.26** 0.13 0.16 23.7

Year 5 OSLERþOSCE sum score 0.29** �0.01 0.17 0.17 11.2 0.09 �0.08 0.28** 17.9

Year 5 OSCE empathy score 0.11 0.01 �0.05 0.11 0.0 0.07 0.08 0.21* 1.4

Years 4 & 5 – penalty points

Year 4 OSLER PPs 0.00 0.17 �0.01 0.11 0.0 �0.12 �0.20 �0.06 0.0

Year 4 OSLER communication PPs 0.09 0.14 �0.04 0.28** 6.8 �0.13 �0.06 0.08 6.0

Year 5 OSCE PPs �0.07 �0.31** �0.04 �0.06 7.6 �0.10 �0.17 �0.18 9.6

Year 5 OSCE communication PPs �0.06 �0.13 �0.14 �0.03 0.0 �0.05 �0.13 �0.21* 0.2

Year 5 OSLER PPs �0.24* 0.07 �0.18 �0.29** 15.6 �0.08 �0.07 �0.23* 18.2

Year 5 OSLER communication PPs �0.18 0.10 �0.26* �0.18 11.2 �0.24* �0.03 �0.15 15.0

Year 5 Clinical exam PPs �0.21* �0.01 �0.39** �0.08 16.3 �0.06 �0.16 �0.23* 19.7

Year 5 OSCE Empathy PPs �0.10 �0.01 0.01 �0.10 0.0 0.06 0.03 0.13 0.0

Year5 OSLERþOSCE sum PPs �0.20* �0.15 �0.21* �0.16 12.2 �0.11 �0.15 �0.27** 17.6

Cells in columns 2–5, 7–9 contain standardised coefficients; cells in columns 6 and 10 give percentage of variance accounted for.

Statistical significance: *50.05 **50.01 (t-test, two-tailed), N¼131–146. Abbreviations as elsewhere.

J. Adam et al.

10

Med

Tea

ch D

ownl

oade

d fr

om in

form

ahea

lthca

re.c

om b

y U

nive

rsity

of

New

cast

le o

n 02

/10/

15Fo

r pe

rson

al u

se o

nly.

Prior academic achievement was related to written Year 4

(but not Year 5) exams, and also to Year 5 (but not Year 4)

clinical exams and penalty marks.

Being younger, or female, or having UK citizenship were

each associated with better performance on many outcomes.

Cognitive reasoning ability, as assessed by the UKCAT

tests, was related to the Year 5 written exam and OSCE

performance.

Non-cognitive variables were not significantly related to

written exams, but were shown to be related to performance in

clinical examinations and other aspects of the course in a

number of ways, both positive and negative, as summarised

below:� Measures of empathy, emotional non-defensiveness,

confidence and emotional intelligence were related to

Year 5 OSLER performance.� Students with high Year 5 OSLER and OSCE PPs tended to

have low empathy and low confidence, with low self-

esteem and low optimism also associated with low Year 5

OSCE scores.� Conscientiousness, confidence, moodiness, emotional

intelligence and emotional non-defensiveness all differ-

entiated the top 20% of Year 5 clinical examination

achievers from the bottom 20%.� Anti-social tendencies and optimism were related to

leaving the course, while aloofness and (poor) image

management related to FTPPPs.

Two general observations can be made. First, the correl-

ations between the different assessment outcomes within the

HYMS course were generally strong, indicating a high degree

of assessment coherence. Second, our findings suggest that

tutors have considerable insight into their students’ behaviour

which eventually correlates with their academic and profes-

sional performance. Tutor ratings in Years 1 and 2 were

positively related to Year 4 performance in written exams,

Year 5 OSLERs, and graduating with Honours, and negatively

related to leaving the course. This consistency may depend on

HYMS’ use of experienced clinicians as PBL tutors, a role tutors

had chosen to undertake as part of a ‘‘portfolio career’’. The

combination of early examination results and such tutor

assessments could thus be useful for identifying those students

who might benefit from targeted professionalism mentoring as

well as academic interventions later in the course.

The breadth and depth of this study is, to date, unique. The

entire cohort of 146 students was tracked through the course,

94% of whom had completed all the initial non-cognitive tests.

The small size of the cohort is compensated to an extent by the

lack of range restriction in the cognitive and non-cognitive

parameters and the completeness of follow-up. The outcome

data covered not only examination results and tutor assess-

ments, but also evidence of problematic professional behav-

iour ranging from minor to serious, collected under HYMS’

system for monitoring fitness-to-practise. Although the large

number of relationships studied means that about 1 in 20 will

appear significant by chance at the 5% level, their consistency

with each other and with the underlying meanings of the

constructs leaves us confident in the validity of our findings.

This approach has yielded evidence that measures of past

academic performance, cognitive skills, and also personality

traits that reflect emotional engagement with people, plus a

positive disposition, all make weak but significant contribu-

tions to the prediction of a range of both desired and

undesirable outcomes from the medical degree course.

A large percentage of complaints about medical practi-

tioners arise from breakdowns in the doctor’s communication

with patients, relatives or colleagues rather than from failings

in procedural skills (see, e.g. the recent annual reports of the

General Medical Council of Great Britain). Communication is a

key skill for a doctor, but is difficult to quantify in students.

This prompted us, a priori, to devise various ways of

measuring different aspects of students’ communication

abilities using the clinical examination data, by breaking

down the marks into components directly related to commu-

nication skills. The different indices derived from the OSLER

and OSCE examinations were thus designed as important

investigative tools to delve into this difficult area, rather than

an attempt to proliferate statistical indices.

We expected to find strong relationships between prior

academic achievement and outcomes, given that McManus

et al. (2013) found that measures of past academic perform-

ance were the main predictors of future academic perform-

ance. This was indeed the case: prior academic performance

(the HYMS academic score) had significant predictive value for

some outcomes. However, none of the other student selection

parameters used by HYMS (UCAS personal statement, referees’

reports and interview scores) predicted performance in any of

the outcome measures in this study, although of course their

use as selection parameters inevitably introduces significant

range restriction.

In contrast, there was no restriction of range for the

cognitive tests and non-cognitive tests as they were not used to

select applicants into the cohort. The non-cognitive variables

that were found to be significant describe aspects of being able

to relate well to other people: empathy, confidence with

others, not being aloof, being able to manage one’s emotions,

being optimistic and having high self-esteem. This is consistent

with the findings of a number of recent studies (e.g., Lambe &

Bristow 2010, Haight et al. 2012, Koenig et al. 2013, Simpson

et al. 2014), which together suggest that such variables may

provide suitable proxy measures for the desired elements of

the ‘‘character’’ advocated by Smyth (1946).

Negative course outcomes such as leaving the courses or

having three or more FTPPPs were related to high scores on

the trait anti-social tendencies. The questions that contribute

to the anti-social tendencies score reflect a disregard for the

laws and norms of one’s society; perhaps, this predicted a

disregard for, or intolerance of, the rules and norms of the

medical educative experience leading to leaving the course or

earning PPs.

The variables of moral decision making (individual freedom

versus societal rules), and the traits of narcissism, anxiety,

irrational thinking, restraint, and permissiveness were found to

be unrelated to outcome as were the scores of self-discipline

and control. Given the numerous literature reports in both

medical education (e.g., Ferguson et al. 2014) and organisa-

tional psychology research that describe the importance of

conscientiousness and other related traits such as self-control

Predictors of outcome at medical school

11

Med

Tea

ch D

ownl

oade

d fr

om in

form

ahea

lthca

re.c

om b

y U

nive

rsity

of

New

cast

le o

n 02

/10/

15Fo

r pe

rson

al u

se o

nly.

and self-discipline, we had expected to find these traits better

predictors.

The relationships described here differ from those reported

when the cohort had completed Year 2 (Adam et al. 2012). For

example, in the first two years of medical education the

variables narcissism, aloofness and irrational thinking pre-

dicted poorer performance, while conscientiousness, confi-

dence and INVOLVEMENT predicted better performance in

Year 1 and Year 2 OSCEs. UKCAT overall scores and the

decision analysis subtest score also predicted Year 1 and Year

2 overall exam scores. Over the complete medical course (five

years) reported here, only the three UKCAT reasoning tests

and the traits of INVOLVEMENT (confidence plus empathy)

have remained as significant predictors.

The demographic factors co-varied with some of the

predictor variables, for example females producing higher

empathy scores compared to males (as is commonly found in

review, e.g. Munro et al. 2005; Wright et al. 2014) even though

there was no evidence of gender difference in prior academic

achievement. The likelihood that demographic factors (age,

gender, citizenship) were confounding variables is supported

by their accounting for additional variance in examination

outcomes over and above that due to ability and personality

variables (Table 5).

The present study demonstrates that it is possible to identify

attributes and qualities, or combination of attributes and

qualities, which might predict outcomes in the academic and

clinical domains from a medical education. As the ultimate

purpose of our research is to assist medical school selectors,

we have also addressed the question of whether the pre-

existing selection method (scrutiny of application form and

interview), other standardised selection tests of cognitive

abilities and non-cognitive traits, intermediate indicators from

the early medical school years (e.g. tutor evaluations) or

demographic variables can reliably predict those more likely to

achieve desirable versus undesirable outcomes.

In summary, our findings show that the qualities that

contribute to good outcomes from a medical course are

academic ability, reasoning ability and a stable and positive

personality. This study is the first to show that measurement of

non-cognitive traits, in combination with measures of aca-

demic and cognitive ability, can predict desirable (and

undesirable) outcomes from medical education. Refinement

of the specific traits and their measurement, together with

medical schools developing assessments in the non-cognitive

domains of medical education, are now the challenges for

future researchers in medical education. There is, after all, an

ethical imperative to ensure we choose the best candidates to

educate as future doctors.

Notes on contributors

JANE ADAM, MA, PhD, MB, BChir, MPH, FFPHM, was an Associate Dean

for Admissions and a PBL tutor at Hull York Medical School from 2003 to

2011, with a research interest in methods for selecting medical students.

MILES BORE gained his PhD in Psychology in 2002, is a registered

psychologist and a senior lecturer at the University of Newcastle, Australia.

His teaching and research interests are in the areas of psychometrics,

personality, moral orientation and the selection of applicants to health

professional education.

ROY CHILDS, BSc, PGCFE, AFBPsS, is the Managing Director of Team

Focus and a Chartered Occupational Psychologist. He works as a coach,

facilitator, trainer and researcher and his main focus is building sustainable

relationships that enhance well-being and performance. His challenge to

orthodox thinking is embodied in an innovative range of psychometric

instruments.

JASON DUNN studied for a PhD in Human Sciences at the Hull York

Medical School, graduating in 2013. He remains an honorary research

fellow of HYMS.

JEAN McKENDREE, PhD, is a senior lecturer in medical education at Hull

York Medical School. She is a cognitive psychologist whose research

involves applying cognitive science principles to improving educational

theory and practice.

DON MUNRO, PhD, is a former staff member and now conjoint associate

professor in the School of Psychology, University of Newcastle. His

interests are in personality and motivation, psychometrics and selection

testing.

DAVID POWIS, PhD, has been a university teacher of, and researcher in,

physiology and medical education since 1972. He is currently conjoint

professor in the School of Psychology at the University of Newcastle. Over

the past three decades, he has worked particularly in the area of medical

student selection with the aim of establishing fair principles and appropri-

ate strategy for selecting students for health professional courses.

Acknowledgements

We are grateful to Professor Barry Wright for allowing us to use

the simulated patients’ ratings of empathy. We express grateful

thanks also to Emeritus Professor John Cookson and to

Professor Jonathan Bennett both of whom read and com-

mented on earlier drafts of the article.

Declaration of interest: Drs. Bore, Munro and Powis are

joint authors of the Personal Qualities Assessment battery of

tests and receive royalty payments when the PQA is used

commercially. Mr. Childs is the owner/manager of Team Focus

who are the commercial publishers of the RSQ. Dr. Adam was

an unpaid member of the UKCAT executive board from 2005

to 2010. Dr. McKendree and Dr. Dunn report no declarations

of interest.

References

Adam J, Bore M, McKendree J, Munro D, Powis D. 2012. Can personal

qualities of medical students predict in-course examination success and

professional behaviour? An exploratory prospective cohort study. BMC

Med Ed 12:69ff.

Barr DA. 2010. Science as superstition: Selecting medical students. Lancet

376:678–679.

Best J. 1989. The politics of the sand pit. Med J Australia 150:158–161.

Bore M, Munro D, Kerridge I, Powis D. 2005. Selection of medical students

according to their moral orientation. Med Ed 39:266–275.

Bore M, Munro D, Powis D. 2009. A comprehensive model for the selection

of medical students. Med Teach 31:1066–1072.

Campbell EF, Rank BK, Sinclair AJM. 1974. Selection of medical students –

A burning question. Med J Australia 1:785–788.

Childs R. 2012. [Accessed 28 January 2015]. Available from: http://

www.teamfocus.co.uk/tests-and-questionnaires/understanding-motiv-

ation/resilience-scales.php.

Cookson J, Crossley J, Fagan G, McKendree J, Mohsen A. 2011. A final

clinical examination using a sequential design to improve cost-

effectiveness. Med Educ 45:741–747.

J. Adam et al.

12

Med

Tea

ch D

ownl

oade

d fr

om in

form

ahea

lthca

re.c

om b

y U

nive

rsity

of

New

cast

le o

n 02

/10/

15Fo

r pe

rson

al u

se o

nly.

Cooper A, Petrides KV. 2010. A psychometric analysis of the Trait

Emotional Intelligence Questionnaire-Short Form (TEIQue-SF) using

Item Response Theory. J Person Assess 92:449–457.

Ferguson E, James D, Madeley L. 2002. Factors associated with success

in medical school: Systematic review of the literature. BMJ 324:

952–957.

Ferguson E, James D, O’Hehir F, Sanders A. 2003. Pilot study of the roles of

personality, references, and personal statements in relation to perform-

ance over the five years of a medical degree. BMJ 326:429–432.

Ferguson E, Semper H, Yates J, Fitzgerald JE, Skatova A, James D. 2014. The

‘dark side’ and ‘bright side’ of personality: When too much conscien-

tiousness and too little anxiety are detrimental with respect to the

acquisition of medical knowledge and skill. PLOS One 9(2):e88606,

1–11.

Goodenough Committee. 1944. Report of the Interdepartmental Committee

on Medical Schools, HMSO, London, p 106. Cited in Smyth DH 1946.

Gray SA, Deem LP, Straja SR. 2002. Are traditional cognitive tests useful in

predicting clinical success? J Dental Educ 66(11):1241–1245.

Groves M, Gordon J, Ryan G. 2007. Entry tests for graduate medical

programs: Is it time to rethink? Med J Australia 186:120–123.

Haight SJ, Chibnall JT, Schindler DL, Slavin SJ. 2012. Associations of

medical student personality and health/wellness characteristics with

their medical school performance across the curriculum. Acad Med 87:

476–485.

Kelly ME, Regan D, Dunne F, Henn P, Newell J, O’Flynn S. 2013. To

what extent does the Health Professions Admission Test-Ireland

predict performance in early undergraduate tests of communication

and clinical skills? – An observational cohort study. BMC Med Ed 13:

68ff.

Koenig TW, Parrish SK, Terregino CA, Williams JP, Dunleavy DM, Volsch

JM. 2013. Core personal competencies important to entering students’

success in medical school: What are they and how could they be

assessed early in the admission process? Acad Med 88:603–613.

Lambe P, Bristow D. 2010. What are the most important non-academic

attributes of good doctors? A Delphi survey of clinicians. Med Teach 32:

e347–e354.

Lancet [editorial]. 1984. Medical student selection in the UK. Lancet 2:1190.

Lockhart LB. 1981. Why aren’t they choosing the right candidates for

medicine? Lancet 7:546–548.

Lynch B, MacKenzie R, Dowell J, Cleland J, Prescott G. 2009. Does

UKCAT predict year 1 performance in medical school? Med Ed 43:

1203–1209.

McKinley DW, Norcini JJ. 2014. How to set standards on performance-based

examinations: AMEE Guide No. 85. Med Teach 36(2):97–110.

McManus IC, Woolf K, Dacre J, Paice E, Dewberry C. 2013. The academic

backbone: Longitudinal continuities in educational achievement from

secondary school and medical school to MRCP (UK) and the specialist

register in UK medical students and doctors. BMC Med 11:242ff.

Munro D, Bore M, Powis D. 2005. Personality factors in professional ethical

behaviour: Studies of empathy and narcissism. Australian J Psychol 57:

49–60.

Petrides KV. 2009. Technical manual for the Trait Emotional Intelligence

Questionnaires (TEIQue). London: London Psychometric Laboratory.

Poole P, Shulruf B, Rudland J, Wilkinson T. 2012. Comparison of UMAT

scores and GPA in prediction of performance in medical school:

A national study. Med Ed 46:163–171.

Schuwirth L, Cantillon P. 2005. The need for outcome measures in medical

education. BMJ 331:977–978.

Simpson PL, Scicluna HA, Jones PD, Cole AMD, O’Sullivan AJ, Harris PG,

Velan G, McNeill HP. 2014. Predictive validity of a new integrated

selection process for medical school admission. BMC Med Educ 14:86.

Smyth DH. 1946. Some principles in the selection of medical students. BMJ

Sep 14:357–367.

Wright B, McKendree J, Morgan L, Allgar VL, Brown A. 2014. Simulated

patients’ ratings of empathy in medical student final year clinical

examination. Are they useful? BMC Med Educ 14:199.

Predictors of outcome at medical school

13

Med

Tea

ch D

ownl

oade

d fr

om in

form

ahea

lthca

re.c

om b

y U

nive

rsity

of

New

cast

le o

n 02

/10/

15Fo

r pe

rson

al u

se o

nly.